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ABSTRACT 


This  thesis  provides  a  comparative  analysis  of  ship  Operational  Test  and 
Evaluation  (OT&E)  in  the  United  States  Navy  (USN)  and  the  Royal  Australian 
Navy  (RAN).  It  also  reviews  the  acceptance  and  introduction  into  service 
procedures  for  warships  in  both  Navies,  including  the  input  from  OT&E.  This  study 
analyses  USN  and  RAN  Ship  OT&E  organisation,  policy,  and  procedures,  and  then 
compares  and  contrasts  the  two  systems.  The  study  finds  that  the  RAN  OT&E 
system,  although  originally  based  on  USN  OT&E  philosophy,  now  differs  in  the 
importance,  interpretation,  application  and  focus  of  OT&E.  It  concludes  that  to 
achieve  efficient  and  effective  trials  and  acceptance  of  the  new  higher  risk  warships 
currently  under  construction,  the  RAN  OT&E  system  needs  to  be  revised.  A  model 
for  OT&E  in  the  RAN  is  proposed  based  on  the  principles  derived  from  the  USN 
system.  The  recommendations  include  the  initial  conduct  of  OT&E  in  land  based 
test  sites,  followed  by  dedicated  “Whole  Ship”  OT&E  for  the  first  of  class. 
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EXECUTIVE  SUMMARY 


With  new  frigates  and  submarines  under  construction,  the  Royal  Australian 
Navy  (RAN)  has  embarked  on  its  greatest  combatant  shipbuilding  program  since  the 
Second  World  War.  Although  these  combatants  are  being  built  to  proven  designs 
and,  therefore,  are  considered  relatively  low  risk,  they  do  present  a  greater 
operational  risk  than  recent  RAN  combatant  programs.  In  September  1991,  the 
Chief  of  Naval  Staff  (CNS)  expressed  concern  that  the  RAN  may  not  be  capable  of 
adequately  managing  the  comprehensive  and  complex  procedures  necessary  to 
accept  the  ANZAC  ships  and  Collins  class  submarines  into  service.  This  focussed 
attention  on  the  acceptance  procedures,  and  consequently  on  the  Operational  Test 
and  Evaluation  (OT&E),  of  the  new  combatants.  The  United  States  Navy  (USN)  is 
the  world  leader  in  the  development  of  high  technology  weapons  systems  and  have 
extensive  experience  in  their  acquisition,  testing  and  operation.  As  the  RAN  has  a 
similar  philosophical  approach  to  acquisition  and  Test  and  Evaluation  (T&E),  the 
USN  OT&E  system  forms  a  credible  basis  from  which  to  develop  an  RAN  OT&E 
organisation,  policies  and  procedures  capable  of  adequately  assessing  the 
operational  effectiveness  and  operational  suitability  of  the  new  combatants  against 
the  user  requirement. 

A.  SHIP  TEST  AND  EVALUATION 

The  fundamental  purpose  of  T&E  in  a  system's  development  and  acquisition 
program  is  to  identify  the  areas  of  technical  risk  to  be  reduced  or  eliminated.  During 
the  eariy  phases  of  development,  T&E  is  conducted  to  demonstrate  the  feasibility  of 
conceptual  approaches,  to  minimise  design  risk,  to  identify  design  alternatives,  to 
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compare  and  analyse  trade-offs,  and  to  estimate  operational  effectiveness  and 
suitability.  As  a  system  undergoes  design  and  development,  the  emphasis  in  testing 
moves  gradually  from  developmental  test  and  evaluation,  which  is  concerned 
chiefly  with  the  attainment  of  engineering  design  goals,  to  operational  test  and 
evaluation,  which  focuses  on  questions  of  operational  effectiveness,  suitability  and 
supportability.  The  principal  types  of  T&E  pertaining  to  ships  as  defined  by  the  USN 
are: 

1.  Developmental  Test  and  Evaluation  (DT&E) 

DT&E  is  conducted  to  assist  the  engineering  design  and  development 
process  and  to  verify  attainment  of  technical  performance  specifications  and 
objectives. 

2.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  testing  conducted  on  production  items  to  ensure  systems  meet 
contract  specifications  and  requirements,  usually  for  contractual  acceptance 
purposes.  It  is  a  type  of  DT&E. 

3.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  conducted  to  determine  a  system’s  operational  effectiveness  and 
operational  suitability,  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  OT  thresholds,  and  develop  tactics.  OT&E  has 
three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment. 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 

•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 
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The  focus  of  DT&E  is  on  a  system  meeting  technical  and  production 
specifications,  while  in  OT&E,  the  focus  is  on  assessing  the  actual  functioning  of  the 
system  in  the  realistic  combat  environment,  against  the  user  requirement. 

R  OT&E  ENVIRONMENT 

The  U.S.  places  a  high  priority  on  Defence  with  strategies  of  world  leadership, 
global  influence  and  self  sufficiency.  To  achieve  these  strategies,  the  U.S.  devotes  a 
high  percentage  of  its  Federal  expenditure  to  Defence,  and  places  a  high  priority  on 
Research  and  Development  (R&D).  The  USN  need  for  OT&E  stems  from  its  almost 
exclusive  reliance  on  indigenous  weapons  development  and  production.  It  has  a 
history  of  large,  risky,  developmental  programs  which  push  the  state  of  the  art  in 
weapons  technology.  To  minimise  technical  and  operational  risk,  the  outputs  of 
these  programs  require  the  assessment  of  operational  effectiveness  and  suitability 
against  the  user  requirement  before  being  committed  to  production  and  subsequent 
introduction  into  the  Fleet. 

Australia’s  Defence  strategy  has  progressed  from  a  position  of  dependence  on 
allies,  to  a  positive  acceptance  of  both  self-reliance  and  regional  influence.  To 
achieve  its  Defence  goals,  however,  Australia  devotes  much  less  of  its  resources  and 
places  less  priority  on  R&D  than  does  the  U.S.  The  need  for  local  T&E  is  recognised 
by  the  Australian  Government  and  DoD,  for  both  indigenous  and  overseas  systems, 
however,  the  importance  erf  OT&E  in  particular,  is  not  specifically  addressed  in  any 
Australian  T&E  policy  document.  This  lack  of  recognition  of  the  importance  of 
OT&E  is  perhaps  due  to  the  RAN  procuring  low  risk,  complete  ships  in  the  past 
(e.g.,  FFG-7  class)  with  the  wealth  of  USN  DT&E  and  OT&E  behind  it. 
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C  OT&E  IN  THE  ACQUISITON  PROCESS 

The  U.S.  Defence  system  acquisition  life  cycle  consists  of  progressive 
development  phases  separated  by  major  decision  milestones,  when  a  program  is 
reviewed  and  authorised  to  advance  to  the  next  stage.  The  USN  OT&E  system  forms 
an  integral  part  of  the  U.S.  DoD  acquisition  process.  It  is  an  empirical  method  of 
ensuring  a  sufficient  technical  return  on  acquisition  investment  and  for  ensuring  that 
a  new  system  is  fully  capable  of  meeting  the  fleet’s  needs.  As  a  result,  the  U.S.  place 
high  importance  on  OT&E,  with  its  conduct  being  mandated  by  Congress  and 
incorporated  in  the  law  of  the  U.S.  OT&E  is  viewed  as  being  more  important  as  a 
basis  for  a  decision  to  proceed  beyond  Low  Rate  Initial  Production,  than  on  the 
introduction  into  service  of  the  final  production  item.  This  is  because  the  “big 
bucks”  of  most  E)efence  acquisitions  are  spent  during  the  production  phase.  OT&E, 
however,  has  started  to  play  an  important  role  also  in  the  assessment  of  first  of  class 
systems  for  introduction  into  the  USN  Fleet 

The  USN  and  RAN  have  similar  acquisition  systems,  however,  the  level  of 
importance  of  T&E,  and  OT&E  in  particular,  differs  greatly.  The  Australian  DoD 
prefers  lower  risk,  proven  technologies  for  the  majority  of  new  acquisitions.  The 
Australian  system,  therefore,  places  a  high  T&E  priority  on  the  post  production 
phase,  with  no  mandatory  requirement  for  OT&E  prior  to  a  production  or  purchase 
decision,  but  requiring  OT&E  before  the  first  production  system  is  Accepted  into 
Naval  Service  (AINS).  As  Australia  is  now  buying  far  less  “off  the  shelf’  systems 
from  overseas,  and  systems  are  increasingly  being  designed  and  built  in  Australia  or 
adapted  from  overseas  designs,  the  operational  risk  of  these  systems  is  increasing. 
These  greater  risks  place  emphasis  on  OT&E  leading  to  a  production  or  purchase 
decision  where  the  majority  of  a  project’s  funding  is  spent. 
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D.  OT&E  ORGANISATION 

The  organisation  of  OT«&E  within  the  USN  is  complex  with  separate  authorities 
performing  a  number  of  OT&E  functions,  as  listed  in  Table  1 . 


TABLE  1 :  USN  OT&E  FUNCTIONAL  AUTHORITIES 


OT&E  Function 

USN  Authority 

Policy  Maker  &  Overseer 

Director,  Operational  Test  &  Evaluation 
(DoD  authority) 

Sponsor 

Chief  of  Naval  Operations 

Developing  Agency 

Commander, 

Naval  Sea  Systems  Command 

Operational  Tester  &  Evaluator 

Commander, 

Operational  Test  &  Evaluation  Force 

User 

Fleet  Commanders, 

Ships’  Commanding  Officers 

Cocffdinator 

Director,  Test,  Evaluation  and  Technology 
Requirements 

Significant  characteristics  of  this  organisation  include  the  appointment  of  a 
Director  of  OT&  E  (DOT&E)  responsible  for  policy  formulation,  evaluation  and 
oversight  of  OT&E  within  the  U.S.  DoD,  and  the  inclusion  of  a  coordinator  for  USN 
T&E.  The  USN  also  separates  the  OT&E  and  DT&E  organisation  structures. 

Following  a  number  of  recent  reorganisations,  the  RAN’s  OT&E  structure  is  in 
disarray.  There  is  no  clear  OT&E,  or  T&E,  policy  maker  or  overseer  within  the 
Australian  E)oD  or  the  RAN.  The  Sponsor  is  not  within  the  RAN  organisation,  being 
part  of  centralised  force  development  and  user  requirements  organisation  (HQADF) 
who  sponsors  major  RAN  acquisitions,  and  is  responsible  for  the  initial 
determination  of  T&E  requirements.  The  Developing  Agency  (ACMAT-N),  unlike 
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its  USN  counterpart,  does  not  have  a  T&E  office  to  support  Project  Directors.  With 
the  disestablishment  of  the  RAN  Trials  and  Assessing  Unit  (RANTAU)  in  May 
1992,  the  RAN’s  OT&E  authority  is  now  the  Commander,  Test  and  Evaluation 
(CTE)  under  the  Chief  Staff  Officer  (Engineering)  (CSO(E))  in  the  Maritime 
Command.  CTE  is  currently  multifunctional,  conducting  DT&E  and  PAT&E  in 
addition  to  OT&E.  This  results  in  a  lack  of  differentiation  within  CTE’s  organisation 
between  DT&E/PAT&E  and  OT&E.  Also  being  placed  under  the  in-service 
engineering  area  of  the  Maritime  Command,  he  lacks  visibility  and  influence  within 
the  warfare  community  and  hence  credibility  in  operational  matters  critical  to 
OT&E.  CTE’s  OT&E  responsibilities  are  overshadowed  by  his  DT&E/PAT&E 
duties.  The  RAN  currently  has  no  single  authority  who  acts  as  a  T&E  focal  point 
within  Navy  Office  responsible  for  coordination  of  T&E  matters,  however,  a  project 
T&E  manager  does  perform  this  function  for  his  project. 

In  summary,  the  RAN  and  Australian  DoD  have  reorganised  over  the  past  five 
years  to  create  a  more  effective  and  efficient  user  requirements  organisation,  and  to 
better  accord  with  Program  Management  and  Budgeting  principles.  However,  as  a 
result,  the  RAN  has  no  T&E,  or  OT&E  policy  maker  and  overseer,  no  OT&E 
coordinator  within  Navy  Office,  and  the  RAN’s  OT&E  authority  is  buried  within  the 
in-service  engineering  management  area  of  the  Maritime  Command. 

E.  OT&E  POLICY 

Where  USN  OT&E  policy  has  its  basis  in  the  statutes  approved  by  Congress, 
the  RAN  has  no  such  formal  basis  and  its  OT&E  policy  is  embedded  in  the  more 
general  T&E  policy.  Some  key  aspects  of  this  policy  include: 
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1.  OT&E  Policy  Documentation 

The  USN  has  a  hierarchy  of  comprehensive  OT&E  policy  documentation 
that  provides  clear,  consistent  and  non-conflicting  policies  and  guidance  to  OT&E 
participants,  and  establishes  the  disciplined  management  approach  to  OT&E  taken 
by  the  USN.  In  contrast,  the  current  RAN  T&E  policy  documentation  provides 
broad  T&E  policy  only  and,  other  than  defining  OT&E.  provides  little  policy  to 
guide  its  conduct.  The  RAN  has  adapted  the  more  detailed  USN  T&E  definitions  to 
suit  its  own  requirements.  During  this  process,  however,  the  RAN  lost  the 
distinction  between  the  types  of  T&E,  and  often  confuses  OT&E  with  PAT&E/ 
DT&E.  Overall,  RAN  T&E  documentation  is  not  comprehensive,  inconsistent  and 
fails  to  provide  adequate  guidance. 

2.  Independence  of  OT&E  Authorities 

Independence  of  OT&E  authorities  is  the  key  to  the  effectiveness  of  OT&E 
in  the  U.S.  acquisition  process.  OPTEVFOR,  as  the  independent  OT&E  agency 
within  the  USN,  reports  directly  to  CNO,  while  DOT&E  provides  independent 
oversight  and  coordination  of  the  military  services’  planning  and  execution  of 
OT&E,  reporting  directly  to  the  Secretary  of  Defence.  The  independence  of  the 
OT&E  authorities  is  designed  to  ensure  impartiality,  and  honest  and  open  reporting, 
subject  to  the  minimum  of  political  interference. 

Although  major  acquisition  decisions  within  the  RAN  are  less  based  on  the 
results  of  OT&E,  the  RAN  also  maintains  the  policy  that  OT&E  is  to  be  conducted 
by  an  authority  independent  of  the  development  and  production  agencies.  CTE,  on 
the  staff  of  the  Maritime  Commander,  is  independent  from  the  development  and 
production  agencies,  but  not  from  the  end  user,  as  in  the  USN.  The  implications  of 
this  lack  of  independence  is  that  the  end  user  may  have  his  own  aims  and  objectives 
which  conflict  with  the  total  impartiality  of  OT&E  conduct  and  reporting. 
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3.  Foreign  Weapons  Evaluation 

The  USN  has  a  foreign  weapons  evaluation  program  which  is  designed  to 
support  the  evaluation  of  foreign  weapons  systems,  equipment  or  technology  in 
terms  of  its  potential  to  meet  specific  U.S.  military  requirements.  The  primary 
objective  of  the  program  is  to  reduce  the  costs  of  research  and  development,  while 
leading  to  the  acquisition  of  foreign  equipment  for  U.S.  use.  Despite  the  most  recent 
Australian  Government  “White  Paper”  on  Defence  Policy  requiring  the  need  to  be 
able  to  determine  the  performance  in  the  Australian  environment  of  equipment  of 
both  overseas  and  local  origin,  no  DoD  or  RAN  T&E  policy  specifically  addresses 
the  evaluation  of  foreign  systems. 

4.  Land  Based  Test  Sites  (LBTS) 

Although  used  primarily  for  the  purpose  of  conducting  DT&E,  the  USN  has 
demonstrated  that  early  OT&E  on  LBTS  can  give  an  estimation  of  potential 
operational  effectiveness  and  suitability,  and  hence  identify  potential  operational 
problems  early  and  minimise  operational  risk.  RAN  guidance  for  OT&E  using 
LBTS  includes  a  statement  that  system  centres  and  simulators  will  be  employed  for 
early  stages  of  OT&E  if  available,  and  that  OT-2  may  be  conducted  at  a  land-based 
test  site.  The  use  of  LBTS  for  OT&E  during  the  AN2^C  frigate  and  COLLINS 
submarine  projects  would  appear  to  dfer  advantages  to  the  RAN.  However  OT-2  in 
support  of  a  production  decision  was  not  conducted  for  either  the  ANZAC  frigate 
and  Collins  submarine  project.  Although  LBTS  are  being  developed  for  both 
projects,  they  were  not  available  before  production  contracts  for  all  the  required 
combatants  were  signed.  The  use  of  these  LBTS  for  eariy  OT-3  also  was  not  planned 
in  the  original  project  schedules,  and  so  the  conduct  of  OT&E  is  now  subject  to  them 
being  used  on  a  non-interfering  basis  to  the  contractor.  LBTS  should  be  used  for 
OT&E  in  the  RAN  to  give  an  estimation  of  potential  operational  effectiveness  and 
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suitability,  and  hence  identify  potential  operational  problems  early  and  minimise 
operational  risk.  This  should  be  conducted  as  the  first  phase  of  OT-3  during  a 
dedicated  period  following  the  completion  of  DT&E  on  the  LBTS. 

5.  Clear  Delineation  between  T&E  Types 

The  USN  achieves  a  clear  delineation  between  the  different  types  of  T&E. 
OPTEVFOR  conduct  only  OT&E,  and  OT&E  within  the  USN  is  conducted  solely 
by  OPTEVFOR.  Thus,  there  appears  to  be  good  understanding  in  the  USN  of  what 
is,  and  what  isn’t,  OT&E.  USN  policy  also  states  that  combined  DT&E/PAT&E  and 
OT&E  testing  should  only  be  considered  when  there  are  time  and  cost  savings.  In 
addition,  this  combined  approach  must  not  compromise  either  the  developmental  or 
operational  test  objectives,  and  separate  evaluations  and  reports  will  be  prepared  by 
DT&E  and  OT&E  testers.  The  conduct  of  combined  and  concurrent  DT&E, 
(PAT&E)  and  OT&E  is  not  addressed  in  RAN  T&E  policy,  however,  it  is  implied  as 
being  almost  a  requirement.  The  RAN  does  not  clearly  differentiate  between  the 
types  of  T&E,  which  leads  to  DT&E  and  OT&E  often  being  conducted  over  the 
same  period,  by  the  same  test  team.  Although  the  RAN  recognises,  by  definition,  the 
difference  in  objectives  and  methodology  between  the  types  of  T&E,  no  limitations 
or  guidance  as  to  the  possible  hazards  of  this  combined  testing  approach  are 
addressed. 

F.  SfflP  ACCEPTANCE 

The  purpose  of  ship  acceptance  in  the  USN  is  to  ensure  delivery  to  the  Fleet  of 
complete  ships,  free  from  both  contractor  and  government  responsible  deficiencies. 
Independent  verification  of  readiness  of  ships  for  acceptance  is  the  responsibility  of 
the  President,  Board  of  Inspection  and  Survey  (PREINSURV)  who  conducts 
Acceptance  Trials  and  Final  Contract  Trials  for  each  individual  new  ship.  Ship 
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acceptance  has  traditionally  been  based  only  on  the  successful  completion  of 
PAT&E  and  material  inspections  that,  for  the  first  of  class,  culminated  in  approval 
for  Fleet  Introduction.  However,  with  the  treatment  of  ships  as  a  complete  weapons 
system,  as  for  the  DDG-51,  Fleet  Introduction  is  now  based  also  on  OT&E  results. 

In  the  RAN,  Delivery  (contractual  acceptance)  of  the  ship  to  the  Government  is 
a  contractual  matter  and  is  managed  by  the  Project  Director.  This  is  followed  by 
RAN  trials  to  assess  the  ship  for  Acceptance  into  Naval  Service  (AINS).  CTE’s 
responsibility  is  to  conduct  the  trials  and  report  the  results.  An  Acceptance  Board 
(AB)  is  established  to  provide  an  assessment  of  the  ship  and  submit 
recommendations  upon  which  CNS  can  base  his  acceptance  decision.  The  President, 
Vice-President  and  Board  members  serve  on  the  AB  part  time,  and  the  sole  function 
of  the  AB  is  to  advise  CNS  on  acceptance  matters.  It  does  not  conduct  tests,  trials  or 
inspections,  and  is  essentially  a  board  of  review.  Similar  to  the  USN,  the  RAN  also 
places  emphasis  on  PAT&E  leading  to  AINS,  although  OT-3  has  been  included  in 
the  trials  program.  However,  the  trials  conducted  as  OT-3  were  really  PAT&E  / 
DT&E  with  minimal  true  OT&E  content.  For  the  first  of  class,  OT-4  is  conducted 
following  AINS.  It  comprises  a  series  of  trials  and  evaluations  over  a  12  to  24  month 
period  to  assess  operational  effectiveness  and  operational  suitability  leading  to 
Operational  Acceptance. 

The  AB,  with  its  broader  experience  and  knowledge  than  CTE,  is  generally 
agreed  within  the  RAN  as  being  successful  as  a  board  of  review  of  the  planning  and 
results  of  T&E  for  individual  projects,  and  recommending  the  first  of  class  for 
AINS.  As  an  ad-hoc  organisation  consisting  of  various  functional  specialists 
brought  together  part  time,  however,  the  AB  suffers  from  a  number  of  problems.  It 
requires  guidance  as  to  its  role  and  functions,  T&E  philosophy  and  the  procedures 
of  the  acceptance  process.  Although  the  AB  attempts  to  obtain  sufficient  guidance 
from  the  T&E  documentation,  it  has  found  this  documentation  to  be  unclear  and 
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conflicting.  With  only  DT&E/PAT&E  and  the  minimal  operational  aspects  of  the 
OT-3  testing  conducted  by  the  RAN,  the  AB  has  tended  to  focus  on  materiel 
deficiencies,  rather  than  the  performance  of  the  first  of  class  against  the  user 
requirement. 

G.  OT&E  PROCEDURES 

1.  Level  and  Extent  of  OT&E 

The  determination  of  the  level  and  extent  of  OT&E  required  is  a  difficult 
question.  The  fundamental  cornerstone  of  OT&E  is  the  user  requirement,  generated 
through  a  series  of  documents  built  into  the  acquisition  process.  A  requirement  can 
be  traced  from  the  capability  need  through  the  refining  requirements  documentation 
to  a  specific  objective  in  the  TEMP,  to  the  test  plan,  and  ultimately  to  the  final  OT&E 
report.  This  linkage  from  capability  need  to  operational  test  program  defines  the 
scope  of  testing  necessary  to  evaluate  the  final  product  against  the  current  user 
requirement. 

2.  Assessing  Operational  Effectiveness 

Operational  effectiveness  is  the  capability  of  a  system  to  perform  its 
intended  function  effectively  over  the  expected  range  of  operational  circumstances, 
in  the  expected  environment,  and  in  the  face  of  the  expected  threat,  including 
countermeasures  where  appropriate.  The  USN  examines  each  Critical  Operational 
Issue  (COI)  and  decides  what  needs  to  be  known  to  enable  each  issue  to  be  assessed. 
It  ensures  that  the  appropriate  environments,  threats,  etc.  are  included  and  that 
sufficient  data  will  be  generated  to  address  the  COI  and  objectives.  It  focuses  on 
achieving  statistical  relevance  and  making  the  tests  as  objective  as  possible.  The 
broad  scope  of  many  operational  requirements,  however,  makes  their  testing  and 
subsequent  assessment  rather  subjective,  making  the  determination  of  meaningful 
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and  assessable  quantitative  Measures  of  Effectiveness  (MOEs)  difficult.  In  addition, 
a  lack  of  resources  may  reduce  an  objective,  quantitative,  statistically  relevant  test 
schedule  to  a  more  subjective,  qualitative  assessment.  These  situations  require 
expertise  and  judgement,  and  compromise  between  the  authorities  involved,  to 
make  the  tests  as  objective  as  possible. 

The  RAN  has  found  it  difficult  to  define  exactly  what  is  required  to  assess 
operational  effectiveness.  The  RAN  recognises  that  the  user  requirement  is  the 
bench  mark  for  determining  the  degree  to  which  a  system  is  effective,  and  also  that 
operational  effectiveness  is  best  assessed  by  a  performance  demonstration  by 
normal  operating  personnel  in  the  normal  or  given  environment.  Assessing 
operational  effectiveness  has  involved  analysing  each  and  then  employing 
modified  Ship  Qualification  Trial  or  Fleet  Exercise  Frogram  techniques  to  evaluate 
them.  The  RAN  has  very  little  guidance  on  the  assessment  of  operational 
effectiveness. 

3.  Assessing  Operational  Suitability 

Operational  suitability  is  the  capability  of  a  system,  when  operated  and 
maintained  by  typical  fleet  personnel  in  the  expected  numbers  and  of  the  expected 
experience  level,  to  be  reliable,  maintainable,  operationally  available,  logistically 
supportable  when  deployed,  compatible,  interoperable  and  safe.  The  assessment  of 
operational  suitability  in  the  USN  consists  of  an  Integrated  Logistic  Support  (ILS) 
Plan  review,  followed  by  suitability  testing  based  on  the  expected  reliability,  degree 
of  confidence,  thresholds,  etc.,  required  to  be  achieved.  OPTEVFOR  has  analysts 
who  design  suitability  tests  and  determine  measures  of  suitability,  and  also  evaluate 
and  analyse  the  adequacy  of  logistic  supportability.  OPTEVFOR  also  relies  on 
operational  personnel  to  use  their  experience  and  knowledge  of  the  system  to 
identify  inadequate  logistic  support. 
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Before  1987,  the  assessment  of  operational  suitability,  and  ILS  in 
particular,  was  rather  subjective  and  often  controversial  in  the  RAN.  A  new 
approach  was  taken  in  1987  to  make  the  assessment  of  ILS  more  objective.  The 
RAN  is  currently  developing  a  routine  in-service  RM«&A  data  collection  and 
analysis  system,  however,  until  that  is  operational,  the  data  is  collected  and  analysed 
on  an  ‘as  required’  basis  by  CTE. 

In  summary,  the  principles  of  operational  suitability  assessment  are 
common  between  the  USN  and  RAN.  Although  assessments  by  the  RAN  have 
become  more  subjective  over  recent  years,  the  assessment  of  RM&A  is  still  in  its 
infancy.  The  RAN  requires  experienced  personnel  to  design  suitability  tests  and 
analyse  results. 

4.  OT&E  Planning 

The  RAN  has  embraced  the  U.S.  TEMP  concept  as  the  single  executive 
document  for  the  management  of  T&E  for  major  acquisitions.  Although  required 
early  in  a  project,  recent  RAN  TEMPs  have  not  been  raised  until  after  project 
funding  has  been  approved.  TEMPS  need  to  be  approved  earlier  to  enable  inclusion 
of  post  delivery  PAT&E/DT&E  and  OT-3  in  the  Project  budget,  and  OT-4  in  the 
Maritime  Commander’s  budget.  The  authority  for  DCNS  to  release  the  TEMP  was 
based  on  his  role  as  Project  Sponsor.  With  HQADF  now  assuming  this  role,  the 
DCNS  function  in  TEMP  development  is  unclear.  The  TEMP  approval  process 
requires  review. 

The  USN  T&E  Coordinator  responsibilities  include  the  chairmanship  of  the 
Test  and  Evaluation  Coordination  Group  (TECG)  for  each  major  program.  Some  of 
the  functions  of  a  TECG  are  the  early  definition  of  terms,  measures  erf  effectiveness, 
and  the  acceptability  criteria.  In  the  RAN,  a  T&E  Planning  Group  (TEPG)  is  formed 
prior  to  TEMP  development  to  analyse  T&E  requirements  and  estimate  the 
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resources  required.  The  TEPG  is  similar  to  the  USN  TECG  with  one  significant 
difference.  The  USN  TECG  is  chaired  by  the  USN  T«&E  Coordinator  under  the 
CNO,  whereas  the  RAN  TEPG  is  chaired  by  each  individual  Project  Director 
through  his  T&E  manager.  Despite  their  best  intentions.  Project  Directors  are 
essentially  driven  by  cost  and  schedule  considerations,  not  the  overall  T&E 
adequacy  of  their  project.  To  be  truly  objective,  each  TEPG  should  be  chaired  by  the 
authority  responsible  for  ensuring  the  adequacy  of  the  RAN’s  overall  T&E  program. 

5.  RAN  OT&E  Reporting 

Reports  provide  the  OT&E  authority’s  conclusions  regarding  a  system’s 
operational  effectiveness  and  suitability,  and  his  recommendations  regarding  the 
systems  future.  The  USN  requires  OT&E  reports  to  be  impartial,  complete, 
thorough  and  be  reported  solely  by  the  OT&E  authority. 

Within  the  RAN,  the  OT&E  Authority  coordinates  the  issue  of  trials 
reports,  however,  they  are  issued  on  completion  of  each  segment  of  OT&E  e.g.,  o/c 
ASW  section  of  OT-3  etc.,  usually  in  addition  to  a  report  at  the  end  of  the  OT  phase. 
Quicklook  reports  are  also  routinely  sent  on  completion  of  each  week’s  testing.  The 
ship  under  test  also  provides  a  report,  however,  it  is  usually  forwarded  to  its 
operational  authority  and  the  Project,  in  addition  to  the  OT&E  Authority.  This  early 
dissemination  of  OT&E  results,  before  a  full  analysis  is  complete  and  the 
implications  assessed,  can  lead  to  other  authorities  taking  hasty  action  on 
incomplete  information,  and  can  prejudice  the  OT&E  authority’s  final  conclusions 
and  recommendations.  OT&E  results  should  indicate  system  deficiencies  against 
user  requirements  rather  than  equipment  defects,  so  the  results  need  to  be  fully 
analysed  and  the  implications  assessed  before  being  reported  to  a  wider  audience. 
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6.  Whole  Ship  OT&E 

The  OT-III  conducted  on  DDG-51  was  the  first  time  the  USN  has 
conducted  OT&E  on  a  whole  ship  in  a  multi-battle  situation,  in  a  free  play 
environment.  The  whole  ship  was  viewed  as  an  integrated  weapons  system,  rather 
than  as  a  platform  for  weapons  systems,  and  OT-III  was  conducted  in  a  dedicated 
period  following  Ship  Qualification  Trials  (SQT)  and  Final  Contract  Trials. 

Within  the  RAN,  the  current  OT&E  to  support  an  AINS  decision  (OT-3)  is 
little  more  than  an  extended  SQT  period  where  DT&E/PAT&E  is  conducted  with 
some  operational  assessment.  It  is  not  considered  to  be  adequate  to  assess  the  first 
of  a  new  ship  class  against  the  user  requirement.  To  assess  the  operational 
effectiveness  and  suitability  of  a  ship  against  the  user  requirement  with  the  degree 
of  confidence  required  for  the  Acceptance  Board  to  support  an  AINS  decision,  a 
dedicated,  free  play,  scenario  based,  OT-3  period  is  required. 

a.  OT&E  Training 

OT&E  is  a  specialised  discipline,  with  its  own  philosophy  and 
methodology,  and  so  requires  a  specialist  approach  with  knowledge  and  experience 
to  make  it  effective.  COMOPTEVFOR  conducts  a  three  day  Operational  Test 
Directors’  (OTD)  course  covering  the  major  areas  of  OT&E.  They  also  run  irregular 
segment  courses  which  provide  overviews  or  updates  on  OT&E  subjects  e.g., 
analysis,  test  plan  development  and  threat  updates. 

The  RAN  has  no  training  courses  on  OT&E,  or  on  T&E  in  general.  A 
number  of  officers  have  completed  the  USN  OTD  course  in  recent  years  which  has 
improved  the  knowledge  and  understanding  of  OT&E  within  CTE’s  organisation 
considerably.  These  courses  have  been  arranged  on  an  ad-hoc  basis  through 
overseas  visit  submissions  rather  than  as  pre-requisite  courses  for  particular  billets. 


To  be  effective,  OT&E  requires  experienced  and  knowledgable 
personnel  with  high  professional  credibility  within  their  field  of  expertise.  To 
achieve  this  requires  selection  of  suitable  personnel,  adequate  training  and  good 
guidance  documentation,  and  preferably  a  career  path  where  serving  in  the  OT&E 
Authority  is  seen  as  career  progression  by  such  capable  personnel. 

H.  CONCLUSION 

The  USN  OT&E  system  has,  by  necessity,  developed  into  a  well  organised, 
well  documented  and  effective,  if  complex,  system.  The  RAN  system,  on  the  other 
hand,  suffers  from  confusing  T&E  policy  documentation,  a  weak  OT&E 
organisation  structure  and  a  general  lack  d  OT&E  knowledge  and  appreciation.  The 
determination  of  OT&E  required  has  traditionally  not  been  achieved  early  enough 
in  the  life  of  a  project,  and  so  project  funding  has  not  included  the  provision  for 
OT&E.  The  RAN  OT&E  system,  although  originally  based  on  USN  OT&E 
philosophy,  now  differs  in  the  importance,  interpretation,  application  and  focus  of 
OT&E.  As  a  result  of  organisation  changes,  it  now  lacks  the  ability  to  effectively 
manage  overall  T&E,  let  alone  OT&E,  at  a  time  when  the  operational  risk  of  ship 
projects  is  increasing.  To  achieve  efficient  and  effective  trials  and  acceptance  of  the 
new  higher  risk  warships  currently  under  construction,  the  RAN  OT&E  system 
needs  to  be  revised. 

L  A  RECOMMENDED  MODEL  FOR  OT&E  IN  THE  RAN 

The  OT&E  model  recommended  for  the  RAN  uses  principles  derived  from  the 
USN  system.  By  taking  consideration  of  resource  limitations,  and  the  characteristics 
and  culture  of  the  RAN,  it  establishes  an  effective  OT&E  organisation  which  is 
important  to  the  testing,  evaluation  and  subsequent  acceptance  decision  of  the  new 
combatants. 
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1.  T&E  Definitions 


The  basis  for  this  OT&E  model  are  clear  definitions  of  the  types  and  phases 
of  T&E.  The  current  Australian  definitions  are  amended  and  phases  of  T&E  are 
refined  to  better  suit  the  RAN  environment.  The  following  definitions  of  types  and 
phases  of  T&E  are  proposed  for  the  RAN  system; 

a.  Developmental  Test  and  Evaluation  (DT&E) 

The  DT&E  definition  remains  unchanged  as  follows: 

DT&E  is  conducted  to  assist  the  engineering  design  and  development  process, 
and  to  verify  attainment  of  technical  performance  specifications  and 
objectives. 

DT&E  consists  of  the  three  phases  listed  in  Table  2. 


TABLE  2:  DT&E  PHASES 


DT&E 

Proposed  RAN  Description 

DTr 

Validation  of  design  concept. 

DT-2 

Demonstration  that  (tesign  meets  specifications. 

DT^ 

Demonstration  that  production  meets  required  technical 
characteristics  or  establish  standards  for  first  of  class. 
PAT&E  is  a  form  of  DT-3. 

b.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

The  definition  of  PAT&E  is  revised  to  recognise  that  it  is  conducted  not 

only  during  the  contract  period,  but  also  during  subsequent  RAN  testing; 

PAT&E  is  conducted  on  production  items  to  ensure  systems  meet  technical 
specifications  and  requirements,  and  is  a  type  of  DT&E. 


PAT&E  consists  of  the  seven  phases  listed  in  Table  3. 


TABLE  3:  PAT&E  PHASES 


PAT&E 

Proposed  RAN  Description 

PAT-0 

Design  and  engineering  development  tests 

PAT-1 

Prxxiuction  and  bum-in  tests 

PAT-2 

Environmental  qualification  tests 

PAT^ 

System  development  tests 

PAT-4 

Harbour  testing 

PAT-5 

Sea  testing 

PAT-6 

Certification  and  Qualification  Trials 

c.  Operational  Test  and  Evaluation  (OT&E) 

The  definition  of  OT&E  is  revised  along  USN  lines,  so  that  it  more 

accurately  describes  the  purpose  and  nature  of  OT&E. 

OT&E  is  conducted  to  determine  a  system's  operational  effectiveness  and 
operational  suitability,  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  OT  thresholds,  and  develop  tactics.  OT&E 
has  three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment. 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 

•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 
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OT&E  consists  of  the  five  phases  listed  inTable  4. 


TABLE  4:  OT&E  PHASES 


OT&E 

Proposed  RAN  Description 

OT-l 

Operational  assessment  of  the  development  projXJsal  to  support  Full 

Scale  Engineering  Development  approval. 

OT-2 

Demonstration  of  achievement  of  program  requirements  for  operational 
effectiveness  and  suitability  to  support  a  production  or  purchase  decision. 

OT-3 

Demonstration  of  achievement  of  program  requirements  for  operational 
effectiveness  and  operational  suitability  on  production  of  ship/aircraft/ 
system  to  support  Acceptance  into  Naval  Service.  Includes  limited 
reliability,  maintainability,  availability  and  logistic  supportability 
assessments. 

OT-4 

Demonstration  of  achievement  of  program  requirements  for  operational 
effectiveness  and  operational  suitability  on  production  of  ship/aircraft/ 
system  in  a  multi-force,  multi-threat  environment  to  support  Operational 
Acceptance.  Includes  detailed  reliability,  maintainability,  availability  and 
logistic  supportability  assessments. 

OTS 

In-service  OT&E,  which  could  include  new  applications,  new  tactics, 
revised  threat,  etc. 

2.  T&E  in  the  Acquisition  Process 

To  ensure  the  progressive  assessment  of  operational  effectiveness  and 
operational  suitability  during  the  acquisition  process,  OT&E  is  scheduled  towards 
the  end  of  each  acquisition  phase.  The  proposed  T&E  schedule,  noting  the 
distinction  between  DT&E/PAT&E  and  OT&E,  is  shown  in  Figure  1.  No  change  is 
suggested  to  the  current  relationship  between  OT&E  phases  and  acquisition 
milestones,  however,  the  figure  separates  DT&E/PAT&E  from  OT&E  to  clearly 
differentiate  these  types  of  testing. 
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FtJLL  SCALE 


PROGRAM  ENGINEERING  PRODUCTION  PRELIM  OA 

INITIATION  DEVELOPMENT  APPROVAL  ACCEPT 

approval 


PHASE  I  PHASE!  PHASE!  PHASE  4  PHASES 


Figure  1:  T&E  in  the  Acquisition  Process 


a.  Total  Ship  Test  Program 

The  Total  Ship  Test  Program  (TSTP)  concept,  modified  from  the  U.S. 
system,  is  now  well  established  in  RAN  shipbuilding  projects.  The  proposed  T&E 
phases  can  be  incorporated  into  an  amended  TSTP.  However,  instead  of  being 
grouped  together  as  in  the  current  TSTP,  the  DT&E  and  OT&E  events  are  now 
shown  as  separate  categories  of  testing.  Because  these  separate  DT&E,  PAT&E  and 
OT&E  events  are  integrated  within  the  overall  TSTP,  these  trials  are  termed 
collectively  as  “Integrated  Tests  and  Trials”,  which  is  an  extension  of  the  current 
RAN  application  <rf  this  term.  Table  5  shows  the  proposed  revised  categories  of  ITT. 
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TABLE  5:  CATEGORIES  OF  ITT 


I 


ITT  Part 

ITT  Category 

T&ET>pe 

Description 

ITT  Part  1 

ITT-0* 

PAT-0 

Design  and  engineering  development  tests 

ITT-1 

PAT-1 

Production  and  bum-in  tests 

ITT-2* 

PAT-2 

Environmental  qualification  tests 

ITT-3* 

PAT-3 

System  development  tests 

ITT^ 

PAT4 

Harbour  testing 

ITT-5 

PAT-5 

Sea  testing 

ITT  Part  2 

nT-6 

DT-3/ PAT-6 

Certification  and  Qualification  Trials 

ITT-?* 

ar-3 

Operational  T&E 

ITT  Parts 

ITT-8* 

ar-4 

Follow  on  Operational  T&E 

Note:  *  Only  performed  on  first  of  class 


3.  OT&E  Organisation 

The  current  OT&E  organisation  structure  is  modified  to  enable  effective 
OT&E  to  support  the  acquisition  process.  It  provides  for  authorities  responsible  for 
the  management,  policy  making  and  oversight,  specification  and  conduct  of  OT&E. 
This  proposed  reorganisation  closely  achieves  the  aim  of  DT&E/OT&E  separation, 
and  is  shown  in  Figure  2. 
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Figure  2:  Proposed  RAN  OT&E  Organisation 


Ideally,  the  head  policy  maker  and  overseer  of  OT&E  should  be  within 
HQADF  or  DoD,  along  the  lines  of  the  U.S.  system.  A  shorter  term,  RAN-only 
solution  would  be  to  include  this  function  within  the  revised  Director  General  Naval 
Policy  and  Warfare  (DGNPW)  organisation  under  the  Deputy  Chief  of  Naval  Staff 
(DCNS).  The  Sponsor  of  major  naval  acquisitions  is  the  Director  General  Force 
Development  (Sea)  (DGFD(Sea))  within  HQADF.  To  be  able  to  determine  the 
appropriate  nature  and  extent  of  OT&E  required,  the  Sponsor  needs  access  to  OT&E 
knowledge  and  experience.  A  close  relationship  needs  to  be  maintained  by 
DGFD(Sea)  with  the  OT&E  policy  maker  and  the  OT&E  authority  (Note  dotted 
line).  To  assist  all  RAN  projects  with  T&E  matters,  an  authority  is  required  within 
the  Materiel  Division,  under  the  Assistant  Chief  of  Naval  Staff  -  Materiel)  ACMAT- 
N),  whose  responsibilities  include  assisting  Project  Directors  in  complying  with 
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policies,  incorporating  lessons  learned  from  previous  projects  and  reviewing  the 
T&E  aspects  of  projects  at  major  milestones.  Although  primarily  a  DT&E  policy 
authority,  this  office  would  also  have  an  input  into  the  OT«feE  process  by  reviewing 
the  TEMP,  and  by  advising  projects  how  to  prepare  for  OT&E.  The  Director  General 
Naval  Engineering  Requirements  (DGNER)  is  currently  responsible  for  the  design, 
development,  acceptance  into  service  and  through  life  support  of  ships.  This 
engineering  policy  directorate  appears  to  be  well  suited  to  the  DT&E  policy  and 
review  function. 

The  current  OT&E  Tester  and  Evaluator,  GTE,  within  the  Maritime 
Command  is  multifunctional,  performing  both  DT&E/PAT&E  and  OT&E.  To 
achieve  the  advantages  of  DT&E  and  OT&E  separation,  a  revised  organisation  is 
proposed.  All  CTE’s  current  DT&E/PAT&E  responsibilities,  and  those  already  the 
responsibility  of  CSO(E),  would  remain  with  an  in-service  trials  team  under 
CSO(E).  CTE’s  OT&E  duties  would  be  removed  and  managed  by  a  separate  group, 
Commander,  OT&E  (COT&E),  working  under  an  operational  authority.  As  OT&E 
is  the  assessment  of  operational  effectiveness  and  suitability,  COT&E  should  be 
responsible  to  COMFLOT.  Headed  by  a  Warfare  branch  officer,  COT&E  would 
manage  the  post  delivery  trials  period  for  all  ships  from  the  MHQ  perspective.  He 
would  still  manage  the  SQT,  which,  although  not  true  OT&E,  still  has  some 
operational  input  which  could  not  be  met  by  the  in-service  trials  team  under 
CSO(E). 

This  proposed  reorganisation  closely  achieves  the  aim  of  DT&E/OT&E 
separation  with  some  concessions  necessary  to  limit  manpower  requirements.  Also, 
by  including  OT&E  managed  by  a  Warfare  Officer  under  COMFLOT,  OT&E  could 
achieve  visibility,  credibility  and  influence  within  the  RAN. 
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4.  RAN  OT&E  Policy 

a.  Policy  Documentation 

To  correct  the  current  inconsistent  and  out  of  date  T&E  documentation, 
a  major  rewrite,  rather  than  just  updating  is  required.  This  rewrite  could  be  the 
vehicle  by  which  the  T&E  system  within  the  RAN  is  overhauled,  as  before  the 
documentation  is  revised,  T&E  definitions  and  terminology  need  to  agreed  within 
the  RAN  and  DoD.  Then  the  OT&E  and  DT&E  organisations  within  the  RAN  need 
to  be  resolved,  and  the  appropriate  policies  decided. 

b.  Independence  of  OT&E  Agency 

Given  the  proposed  COT&E  is  organisationally  independent  from  the 
developing  and  production  agencies,  his  procedural  independence  should  also  be 
maintained  to  ensure  impartiality  and  validity  of  testing.  When  OT&E  is  conducted, 
it  should  not  be  influenced  by  the  Project  or  Design  Approval  Authority  (DAA).  No 
Project  or  DAA  personnel  should  be  present  during  testing,  A  final  report  only 
should  be  issued  by  the  Maritime  Commander,  with  no  progress  or  interim  report 
(unless  the  testing  is  unusually  lengthy),  to  allow  full  analysis  and  evaluation  of  the 
test  results.  Similarly,  to  ensure  test  validity  and  impartiality,  contractors  should  not 
be  present  during  OT&E. 

5.  T&E  Planning 

Test  and  Evaluation  Master  Plans  (TEMPs)  would  be  approved  early  to 
enable  inclusion  of  post  delivery  PAT&E/DT&E  and  OT-3  in  the  Project  budget,  and 
ClT-4  in  the  Maritime  Commander’s  budget.  To  be  truly  objective,  each  TEPG 
would  be  chaired  by  the  authority  responsible  for  ensuring  the  adequacy  of  the 
RAN’s  overall  T&E  program.  The  proposed  RAN  OT&E  policy  maker  and 
overseer,  DGNPW,  may  best  be  suited  to  the  chairmanship  of  each  TEPG. 
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6.  Land  Based  Test  Sites 

OT&E  would  be  conducted  during  a  dedicated  period  in  the  LBTS 
following  successful  DT«feE,  to  gain  an  estimate  of  the  potential  operational 
effectiveness  and  suitability  of  the  system.  This  period  would  be  planned  early  so  as 
to  be  included  in  Project  schedules  and  funds. 

7.  Whole  Ship  Testing 

To  assess  the  operational  effectiveness  and  suitability  of  a  ship  against  the 
user  requirement  with  the  degree  of  confidence  required  for  the  Acceptance  Board 
to  support  an  AINS  decision,  a  dedicated,  free  play,  scenario  based,  OT-3  period  is 
proposed.  This  OT-3  would  be  conducted  after  the  ship  has  completed  the  SQT, 
Workup  and  Operational  Readiness  Evaluation  to  ensure  that  the  testing  is 
conducted  on  a  worked  up  and  materially  proven  ship.  OT-3  would  be  conducted  of 
the  overall  ship  as  a  complete  weapons  system. 

8.  RAN  Ship  Acceptance 

The  AB  would  be  convened  for  the  first  of  class  only,  providing  the 
specialist  expertise  to  interpret  requirements  and  determine  the  level  of  acceptability 
required  for  AINS.  The  AB  consists  of  Naval  Officers  drawn  part-time  from  their 
normal  jobs,  and  so  the  cost  appears  to  be  minimal.  The  AB,  however,  has  a  high 
opportunity  cost,  which  is  the  value  of  the  next  best  alternative  on  which  the 
members  could  be  working  instead  of  serving  on  the  Board.  Once  the  first  of  class 
is  accepted  the  Board’s  task  is  complete.  The  interpretations  and  levels  of 
acceptability  established  by  the  AB  then  guide  the  Trials  Authority  in  recommending 
acceptance  of  each  of  the  remaining  ships.  With  a  more  comprehensive  OT-3  period 
proposed  to  be  conducted,  the  AB  members  would  more  effectively  use  their 
knowledge  and  experience  in  assessing  the  results  of  OT&E  against  the  user 
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requirement  rather  than  on  workmanship  and  materiel  deficiencies.  The  AB  should 
remain  as  a  board  of  review. 

9.  RAN  OT&E  Procedures 

a.  Assessment  of  Operational  Effectiveness 

With  the  proposed  inclusion  of  a  dedicated  OT-3  period,  the  OT&E 
authority  will  need  to  develop  tests  for  a  free  play,  scenario  based  environment 
specifically  designed  to  assess  the  COIs  against  the  User  Requirements.  The 
analysis  of  these  tests  may  be  different  from  the  weapons  analysis  currently 
performed. 

b.  Assessment  of  Operational  Suitability 

The  assessment  of  operational  suitability,  especially  for  the  RAN’s 
current  combatant  projects,  would  focus  on  interoperability  and  compatibility  as 
these  present  the  highest  risks  with  the  integration  of  proven  equipment  in  a  unique 
combination  in  a  ship  platform.  RM&A  data  is  collected  and  analysed  during  OT-3 
to  give  an  indication  of  performance  prior  to  AINS,  and  then  continued  for  12 
months  during  the  OT-4  phase.  The  data  would  continue  to  be  collected  and 
analysed  on  an  '‘as  required”,  rather  than  on  a  continuous  basis  by  the  Trials 
Authority  until  the  RAN  develops  its  routine  in-service  RM&A  data  collection  and 
analysis  system.  The  analysis  of  RM&A  data  is  a  specialist  task,  currently  outside 
the  experience  of  most  uniformed  personnel.  To  achieve  significant  and  valid 
results,  an  improved  RM&A  analysis  capability  would  be  established. 

c.  OT&E  Training 

A  formal  course  such  as  the  USN  Operational  Test  Directors  Course 
should  be  a  pre-requisite  for  key  billets  within  the  OT&E  authority  until  the  RAN 
develops  enough  knowledge  and  expertise  to  develop  its  own  course.  Other 
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authorities  also  require  training  in  OT&E.  The  OT&E  authority  should  develop 
acquaint  courses  for  other  personnel  with  a  need  to  understand  OT&E.  Also  to 
increase  the  awareness  of  OT&E  in  the  RAN,  the  OT&E  authority  would  deliver 
presentations  regularly  to  Project  Directors  Courses,  the  ILS  Management  and 
Acquisition  Course  (ILSMAC),  Defence  management  courses,  and  other 
appropriate  acquisition  courses. 

The  proposed  change  of  the  head  of  the  OT&E  authority  to  a  Warfare 
Officer  as  COT&E  would  provide  a  career  path  within  the  authority.  As  the 
credibility  of  OT&E  increases  within  the  warfare  community,  the  knowledge  and 
experience  gained  by  warfare  officers  in  this  area  will  be  seen  to  be  of  more  benefit 
in  future  warfare  related  positions.  The  proposed  employment  of  COT&E  under 
COMFLOT  should  also  be  more  career  enhancing  for  warfare  personnel  than  the 
current  arrangement  under  CSO(E). 

The  professionalism  and  knowledge  of  T&E  personnel  could  be 
enhanced  by  membership  in  a  related  professional  organisation.  The  corporate 
membership  of  a  suitable  professional  organisation  may  be  worthwhile  to  the  RAN. 

J.  RECOMMENDATIONS 

It  is  recommended  that  the  proposed  OT&E  model  be  reviewed,  evaluated  and 
implemented  in  the  RAN  to  achieve  an  effective  OT&E  system  to  support  the 
introduction  of  operationally  effective  and  operationally  suitable  new  combatants 
into  the  RAN. 

1.  OT&E  Definitions 

It  is  recommended  that: 

a.  the  current  definitions  and  phases  of  ship  T&E  be  amended  to  better  suit 
the  Australian  Defence  environment. 
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b.  the  current  TSTP  be  revised  to  show  DT&E  and  OT&E  events  as 
separate  categories  of  testing,  and  these  trials  be  termed  collectively  as 
Integrated  Tests  and  T rials  (ITT). 

2.  OT&E  Documentation 

It  is  recommended  that; 

a.  the  RAN  T&E  DI(N)  and  ABR  1921  be  revised  to  achieve 
comprehensive  and  consistent  T&E  policy  guidance. 

3.  OT&E  in  the  Acquisition  Process 

It  is  recommended  that: 

a.  OT&E  be  conducted,  where  possible,  prior  to  the  production  or 
purchase  decision  in  the  acquisition  process  to  minimise  risk. 

b.  OT-S  be  separated  from  the  DT&E  /PAT&E  post  delivery  trials. 

c.  C)T-3  be  conducted  as  a  dedicated,  free  play,  scenario  based  period  after 
the  completion  of  SQT  and  ORE  to  ensure  testing  is  conducted  on  a 
worked  up  and  materially  proven  ship. 

d.  OT-3  of  the  overall  ship  be  conducted  as  a  complete  weapons  system. 

4.  OT&E  Organisation 

It  is  recommended  that: 

a.  the  appointment  of  a  DoD  OT&E  policy  maker  and  overseer  within 
HQADF  be  investigated. 

b.  DGNPW  be  appointed  as  the  OT&E  policy  maker  and  overseer  within 
the  RAN. 

c.  a  close  working  relationship  on  T&E  matters  be  required  between 
DGFD(Sea),  DGNPW  and  the  OT&E  authority. 
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d  DGNER  be  appointed  as  the  Materiel  Division  T&E  agency  to  assist 
Project  Directors  in  complying  with  policies,  incorporate  lessons 
learned  from  previous  projects,  review  the  T&E  aspects  of  projects  at 
major  milestones,  and  advise  projects  how  to  prepare  for  OT&E. 

e.  the  current  CTE  organisation  be  disestablished. 

f.  an  “In-service  Trials”  team  be  established  under  CSO(E)  to  conduct 
CTE’s  current  DT&E/PAT&E  responsibilities,  and  those  already  the 
responsibility  of  CSO(E). 

g.  Commander,  OT&E  (COT&E)  be  established  as  the  RAN’s  OT&E 
authority  under  COMFLOT,  to  manage  OT&E  and  the  post  delivery 
trials  period  for  all  ships  from  the  MHQ  perspective. 

5.  OT&E  Policy 

It  is  recommended  that: 

a.  policy  be  developed  to  enable  the  performance  in  the  Australian 
environment  of  foreign  equipment  to  be  evaluated  efficiently, 
effectively  and  consistently. 

b.  the  TEMP  approval  process  be  reviewed  to  ensure  appropriate 
authorities  are  consulted  and  approve  the  TEMP. 

c.  the  TEPG  be  chaired  by  DGNPW  as  the  authority  appointed  to  ensure 
the  adequacy  of  the  RAN’s  overall  T&E  program. 

d.  OT&E  not  be  influenced  by  the  Project  or  DAA,  and  that  Project  or 
DAA  personnel  not  be  present  during  testing. 

e.  contractors  not  be  present  during  OT&E. 

f.  TEMPs  for  non-development  projects  be  approved  before  production 
approval  to  enable  inclusion  of  post  delivery  PAT&E/DT&E  and  OT-3 
in  the  Project  Budget,  and  OT-4  in  MHQ  budget. 
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g.  OT&E  be  conducted  during  a  dedicated  period  in  the  LBTS  following 
successful  DT&E,  to  gain  an  estimate  of  the  potential  operational 
effectiveness  and  suitability  of  the  system. 

6.  Ship  Acceptance 

It  is  recommended  that; 

a.  the  Acceptance  Board  remain  as  a  board  of  review. 

b.  the  Acceptance  Board  be  convened  for  the  first  of  class  only . 

7.  OT&E  Procedures 

It  is  recommended  that: 

a.  The  OT&E  authority  develop  tests  for  a  free  play,  scenario  based 
environment  specifically  designed  to  assess  the  COIs  against  the  user 
requirement. 

b.  analysis  requirements  of  these  tests  be  assessed,  as  they  may  be 
different  from  the  weapons  analysis  currently  performed. 

c.  experienced  personnel  assist  in  designing  suitability  tests  and  analysing 
results. 

d  interoperability  and  compatibility  be  a  major  focus  during  the 
assessment  of  operational  suitability,  especially  for  the  RAN’s  current 
combatant  projects. 

e.  an  improved  RM&A  analysis  capability  be  established. 

f.  RM&A  data  continue  to  be  collected  and  analysed  on  an  ‘as  required’ 
basis  until  the  RAN  develops  its  routine  in-service  RM&A  data 
collection  and  analysis  system 

g.  OT&E  reports  be  made  solely  by  the  OT&E  authority. 

h.  OT&E  phase  reports  be  issued  only,  with  no  progress  or  interim  reports, 
to  allow  full  analysis  and  evaluation  of  the  test  results. 
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i.  the  USN  Operational  Test  Director  type  course  be  a  prerequisite  for  key 
billets  within  the  OT&E  authority  until  the  RAN  develops  enough 
knowledge  and  expertise  to  develop  its  own  course. 

j.  the  OT&E  authority  develop  acquaint  courses  for  other  personnel  with 
a  need  to  understand  OT&E. 

k.  the  OT&E  authority  deliver  presentations  regularly  to  Project  Directors 
Courses,  the  ILS  Management  and  Acquisition  Course  (ILSMAC), 
Defence  management  courses,  and  other  appropriate  acquisition 
courses,  so  to  increase  the  awareness  of  OT&E  in  the  RAN. 

l.  a  Warfare  Officer  be  appointed  to  head  the  OT&E  authority. 

m.  establishing  corporate  membership  of  a  suitable  professional  T&E 
organisation  be  investigated. 
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I.  INTRODUCTION 


A.  BACKGROUND 

1.  Introduction 

Friday,  May  15,  1992  marked  the  end  of  an  era  for  Operational  Test  and 
Evaluation  (OT&E)  in  the  Royal  Australian  Navy  (RAN).  The  RAN  Trials  and 
Assessing  Unit  (RANTAU),  the  RAN’s  Operational  Test  and  Evaluation  authority, 
ceased  to  exist  as  an  independent  unit  and  was  integrated  with  Maritime 
Headquarters  (MHQ)  under  the  Commander,  Test  and  Evaluation.  [Ref.  56]This 
marks  a  continuing  shift  in  the  emphasis  placed  on  OT&E  within  the  RAN. 

2.  Field  of  Study 

The  fundamental  purpose  of  Test  and  Evaluation  (T&E)  in  a  system’s 
development  and  acquisition  program  is  to  identify  the  areas  of  technical  risk  to  be 
reduced  or  eliminated.  Operational  Test  and  Evaluation  is  the  means  to  demonstrate 
that  a  ship  meets  the  specified  user  requirement  before  being  Accepted  into  Naval 
Service  (AINS).  This  study  analyses  USN  and  RAN  Ship  OT&E  organisation, 
methodology,  and  procedures,  and  then  compares  and  contrasts  the  two  systems. 
From  this  analysis,  a  model  for  OT&E  in  the  RAN  is  proposed. 

3.  Importance  of  Research  Effort 

In  September  1991,  the  Chief  of  Naval  Staff  (CNS)  expressed  concern  that 
the  RAN  may  not  be  capable  of  adequately  managing  the  comprehensive  and 
complex  procedures  necessary  to  accept  the  ANZAC  frigates  and  Collins  class 
submarines  into  service  [Ref.  62].  With  these  new  classes  of  surface  and  sub-surface 
combatants  under  construction,  and  the  RAN  currently  reorganising  its  OT&E 
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organisation,  the  RAN  could  benefit  from  a  timely  analysis  of  USN  and  RAN  Ship 
OT&E.  The  study  is  designed  to  assist  the  RAN  to  develop  an  effective  and  efficient 
OT&E  organisation. 

R  OBJECTIVES 

OT&E  is  conducted  to  determine  a  system’s  operational  effectiveness  and 
operational  suitability,  to  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  operational  test  thresholds,  and  to  develop  tactics. 
It  is  usually  conducted  at  the  end  of  each  phase  of  the  acquisition  process  to  support 
a  major  milestone  decision.  The  more  significant  of  these  is  the  OT&E  conducted 
on  a  prototype  to  support  a  production  or  purchase  decision,  and  on  the  first  of  class 
to  support  an  Acceptance  into  Naval  Service  decision  (AINS).  OT&E  is  conducted 
against  the  current  user  requirement  and  is  definitive  feedback  to  the  program’s 
sponsor  that  the  system  has  met  the  user  requirement. 

OT&E  has  a  high  level  of  representation  within  the  U.S.  E>epartment  of  Defence 
and  the  USN.  The  Director,  Operational  Test  and  Evaluation,  has  direct 
responsibility  to  the  Secretary  of  Defence  for  prescribing  policies  and  procedures 
governing  the  conduct  of  OT&E  in  the  Department  of  E>efence,  and  is  a  member  of 
the  Defence  Acquisition  Board.  The  USN  has  a  dedicated  command,  the 
Commander  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
responsible  to  the  Chief  of  Naval  Operations  (CNO)  for  Ship  OT&E. 

The  RAN  is  currently  reorganizing  it’s  OT&E  authority.  Prior  to  May  1992,  the 
RAN  Trials  and  Assessing  Unit  (RANTAU)  was  responsible  to  the  Naval  Warfare 
Branch,  as  the  originator  of  user  requirements,  for  the  planning,  conduct  and 
reporting  of  OT&E.  However,  RANTAU  has  recently  been  disbanded  and  the 
OT&E  component  transferred  to  the  RAN  Maritime  Commander  -  the  eventual  ship 
“user”. 


Problems  were  experienced  in  the  RAN  with  a  perceived  lack  of  high  level 
representation  and  consideration  of  the  OT&E  requirements  during  the  early  phases 
of  programs,  with  the  cost  and  time  effectiveness  of  OT&E.  with  the  conduct  of 
OT&E  and  Developmental  Test  and  Evaluation  (DT&E)  concurrently,  and  with 
differences  between  acceptance  of  an  item  from  a  contractor  (Delivery)  and 
Acceptance  into  Naval  Service. 

The  RAN  is  a  comparatively  small  navy  with  limited  resources.  With  large 
indigenous  programs  including  the  new  Collins  class  submarines  and  ANZAC  class 
frigates  due  to  undertake  OT&E  in  the  near  future,  it  is  imperative  that  the  RAN  has 
a  cost  effective  and  efficient  OT&E  organisation. 

This  study  compares  the  USN  and  RAN  OT&E  systems  and  highlights 
strengths  and  weaknesses  of  each.  The  methodology  and  procedures  currently  used 
by  each  Navy  for  the  assessment  of  both  operational  effectiveness  and  operational 
suitability  are  analysed.  The  lessons  learned  from  OT&E  of  the  most  recent  USN 
ship  program,  the  DDG-51,  and  the  RAN  DDG  Modernization  and  Australian 
Frigate  programs  are  reviewed.  Then,  using  the  results  of  this  analysis,  the  current 
RAN  plan  for  the  OT&E  of  the  ANZAC  class  frigate  and  Collins  class  submarine 
are  reviewed.  The  output  of  the  study  is  a  model  for  the  organisation  and  conduct  of 
Ship  OT&E  in  the  RAN. 

C  THE  RESEARCH  QUESTION 

The  primary  research  question  posed  in  this  thesis  is  “What  are  the  fundamental 
characteristics  of  USN  ship  OT&E  and  how  may  USN  experience  in  OT&E  be 
applied  to  achieve  an  efficient,  effective  OT&E  organisation  for  the  RAN?” 
However,  to  address  the  fundamental  characteristics  of  OT&E,  a  number  of 
subsidiary  research  questions  need  to  be  answered.  Some  of  the  possible  questions 
follow.  How  does  OT&E  support  the  Defence  acquisition  process?  What  are  the 
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fundamental  characteristics  of  the  USN  OT&E  organisation?  What  are  the 
fundamental  characteristics  of  USN  OT&E  policy?  What  are  the  fundamental 
characteristics  of  USN  OT&E  procedures?  How  is  USN  OT&E  implemented?  How 
does  OT&E  support  the  acceptance  of  ships  into  the  USN  Fleet?  How  do  the  USN 
and  RAN  OT&E  systems  currently  compare?  What  needs  to  be  done  in  the  RAN  to 
achieve  an  effective  OT&E  system  with  a  minimum  of  additional  resource 
requirements? 

D.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

The  scope  of  this  thesis  is  limited  to  the  study  of  OT&E  for  naval  warships  and 
submarines.  Although  based  on  the  same  philosophy,  the  OT&E  of  aircraft  and 
equipment  is  not  addressed.  Although  other  nations  conduct  OT&E  (e.g..  United 
Kingdom),  this  study  is  limited  to  the  analysis  of  USN  and  RAN  OT&E  systems 
only.  This  study  is  an  overview  of  OT&E  organisation,  methodology  and 
procedures.  It  does  not  address  specific  implementation  matters  or  details  of 
individual  tests  or  trials.  This  study  is  limited  to  UNCLASSIFIED  material  only, 
and  the  collection  of  RAN  information  was  limited  to  written  and  verbal 
correspondence  as  no  personal  interviews  were  able  to  be  conducted  with  RAN 
authorities.  No  substantial  increase  of  RAN  OT&E  resources  in  the  near  future  was 
assumed. 

E.  WHY  STUDY  USN  OT&E? 

Study  of  the  USN  OT&E  system  could  be  beneficial  to  the  RAN  for  a  number 
of  reasons.  Due  to  the  RAN’s  long  association  with  the  USN  and  its  purchase  of  U.S. 
ships  and  equipment,  it  has  a  similar  philosophical  approach  to  acquisition  and 
T&E.  The  basis  <rf  much  of  the  current  RAN  T&E  policies,  methods  and  procedures 
is  derived  from  the  USN. 
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Technology  is  changing  the  face  of  modem  warfare,  and  the  USN  is  at  the 
leading  edge  of  this  movement,  adapting  the  best  of  new  technologies  to  proven 
concepts  and  platforms.  The  USN  strives  to  ensure  that,  as  technology  evolves,  it  is 
used  to  best  advantage.  As  OT&E  determines  a  new  system’s  operational 
effectiveness  and  operational  suitability,  the  need  for  potential  modifications  to 
meet  established  OT  thresholds,  and  develops  tactics,  it  is  a  critical  part  of  adapting 
technology  to  modem  warfare. 

Being  the  most  modem  and  capable  naval  force  in  the  world  today,  the  USN  has 
substantial  weapons  system  acquisition,  operational  and  combat  experience.  This 
experience  has  confirmed  their  need  for  OT&E,  and  refined  their  OT&E 
organisation,  policies,  procedures  and  methodology.  The  RAN  could  benefit  from 
this  experience. 

The  USN  are  world  leaders  in  the  development  of  high  technology  weapons 
systems  and  have  extensive  experience  in  their  acquisition,  testing  and  operation.  As 
the  RAN  has  a  similar  philosophical  approach  to  acquisition  and  T&E,  the  USN 
OT&E  system  forms  a  good  basis  from  which  to  develop  an  RAN  OT&E 
organisation,  policies,  procedures  and  methodology. 

F.  RESEARCH  METHODOLOGY  AND  LITERATURE  REVIEW 

A  balanced  research  methodology  was  used  following  the  general  research 
guidance  outlined  in  Reference  1 .  The  research  consisted  of  three  strategies: 

1.  Archival  Research 

Archival  research  consisted  of  examining  the  relevant  form''l  documents 
pertaining  to  OT&E  in  both  navies.  This  included  a  review  of  U.S.  Defence 
Acquisition  law,  followed  by  review  of  the  U.S.  DoD  and  USN  official 
documentation,  pertaining  to  T&E,  and  to  ship  OT&E  in  particular.  A  similar 
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approach  was  taken  in  reviewing  the  Australian  documentation.  Also,  a  literature 
search  was  conducted  to  elicit  the  latest  published  views  and  opinions  on  OT&E. 
This  search  was  conducted  through  two  primary  sources; 

a.  Database  Search 

A  search  was  conducted  with  DIALOG  Information  Services,  a 
commercial  information  service  available  through  the  Dudley  Knox  Library,  Naval 
Postgraduate  School.  The  DIALOG  service  is  a  computer  database  accessible  by 
modem.  The  search  was  conducted  using  a  number  of  key  words  including 
Operational  and  Evaluation,  policy,  procedures,  and  destroyer  and  frigate.  A  listing 
of  200  recent  items  was  provided  [Ref.  2]. 

b.  Defence  Logistics  Studies  Information  Exchange 

The  E>efence  Logistics  Studies  Information  Exchange  (DLSIE),  located 
at  the  U.S.  Army  Logistics  Management  College,  Fort  Lee,  Virginia,  is  a  U.S.  DoD 
organisation  that  acquires,  organises,  stores  and  disseminates  logistics  and 
management  information  on  a  DoD  wide  basis  [Ref.  3].  A  custom  bibliography  was 
requested  on  the  subject  of  ship  Operational  Test  and  Evaluation.  A  listing  of  48 
studies  and  reports  on  the  subject  was  provided  [Ref.  4]. 

Other  sources  of  information  included  Journals  of  professional 
organisations  and  Defence  magazines,  however,  the  most  useful  guidance  to 
relevant  publications  was  provided  by  personnel  involved  in  OT«&E.  In  summary, 
the  archival  research  established  the  current,  formal  organisation,  methodology  and 
conduct  of  OT«&E  in  both  navies,  and  identified  the  latest  public  opinions  on  OT&E 
issues. 
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2.  Personal  Interview  Research 

Ship  OT&E  is  a  highly  specialised,  evolving  field  of  expertise.  Many 
developments  are  not  published  formally  by  DoD  or  in  the  commercially  available 
literature.  It  was  necessary,  therefore,  not  only  to  survey  the  literature  on  the  subject, 
but  also  to  conduct  interviews.  This  research  was  conducted  by  personal  interviews, 
either  in  person  or  over  the  phone,  with  key  participants  in  the  ship  OT&E  field. 
Views  and  opinions  were  sought  from  authorities  in  both  navies  who  either  conduct, 
use  the  results  of,  or  are  otherwise  effected  by  ship  OT&E.  The  list  of  pertinent 
references  as  well  as  personnel  interviewed  are  contained  in  the  References  section 
of  this  thesis. 

3.  Research  Analysis 

Research  analysis  was  then  applied  to  the  information  gathered  by  the  other 
two  strategies,  to  assess  the  information,  evaluate  the  strengths  and  weaknesses  of 
each  system,  and  to  develop  the  OT&E  model.  This  analysis  consisted  of  breaking 
down  the  USN  and  RAN  OT&E  systems  into  their  component  parts,  in  order  to 
determine  how  (and  where  possible,  why)  the  systems  are  organised,  and  particular 
procedures  and  methods  used.  An  OT&E  model  for  the  RAN  was  then  developed 
from  the  results  and  conclusions  of  the  research  analysis.  As  the  RAN  has  a  similar 
philosophical  approach  to  acquisition  and  T&E  to  the  USN,  their  OT&E  system 
forms  a  good  basis  from  which  to  further  develop  RAN  OT&E  organisation, 
policies,  procedures  and  methodology. 

G.  DEFINITIONS  AND  ABBREVIATIONS 

Terms  important  to  the  analysis  of  OT&E  systems  in  the  RAN  and  USN  are 
defined  as  they  are  introduced.  A  list  of  acronyms  is  included  in  Appendix  B. 
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H.  ORGANISATION  OF  THESIS 

The  study  initially  reviews  the  background  of  OT&E  in  Chapter  II.  The  purpose 
of  T«feE  in  Defence  acquisition  is  discussed  and  the  types  of  T&E  applicable  to  ships 
are  introduced.  The  U.S.  and  Australian  strategic,  economic  and  Defence 
environments  are  discussed  briefly  and  compared.  Some  similarities  and  differences 
between  the  USN  and  RAN  are  also  noted.  The  philosophy,  organisation  and 
management  of  OT«&E  within  the  USN  and  RAN  are  then  analysed  in  Chapters  III 
and  IV  respectively.  A  comparative  analysis  of  USN  and  RAN  OT«feE  systems  is 
included  in  Chapter  V.  As  result  of  this  analysis,  a  model  for  OT&E  in  the  RAN  is 
proposed  in  Chapter  VI.  Chapter  VII  details  the  conclusions  and  Chapter  VIII  lists 
the  recommendations. 
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II.  OPERATIONAL  TEST  &  EVALUATION  BACKGROUND 

This  chapter  presents  background  information  considered  necessary  for  a  better 
understanding  of  OT&E.  The  purpose  of  T&E  in  Defence  acquisition  is  discussed 
and  the  types  of  T&E  applicable  to  ships  are  introduced.  The  U.S.  and  Australian 
strategic,  economic  and  defence  environments  are  discussed  briefly  and  compared. 
Some  similarities  and  differences  between  the  USN  and  RAN  are  also  noted. 

A.  PURPOSE  OF  T&E  IN  DEFENCE  ACQUISITION 

The  fundamental  purpose  of  T&E  in  a  defence  system’s  development  and 
acquisition  program  is  to  identify  the  areas  of  technical  risk  to  be  reduced  or 
eliminated.  During  the  early  phases  of  development,  T&E  is  conducted  to 
demonstrate  the  feasibility  of  conceptual  approaches,  to  minimise  design  risk,  to 
identify  design  alternatives,  to  compare  and  analyse  trade-offs,  and  to  estimate 
operational  effectiveness  and  suitability.  As  a  system  undergoes  design  and 
development,  the  emphasis  in  testing  moves  gradually  from  developmental  test  and 
evaluation,  which  is  concerned  chiefly  with  the  attainment  of  engineering  design 
goals,  to  operational  test  and  evaluation,  which  focuses  on  questions  of  operational 
effectiveness,  suitability  and  supportability.  [Ref.  18:p.  1-1]  A  primary  contribution 
made  by  T&E  is  the  identification  and  reporting  of  deficiencies  that  may  adversely 
impact  the  performance  capability,  avsulability  or  supportability  of  a  system. 

T&E  serves  a  number  of  useful  functions,  providing  information  for  the 
following  customers: 

•  Developers  to  assist  in  the  identification  and  resolution  of  technical 
difficulties. 
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•  Decision  makers  responsible  for  making  the  investment  decision  to  procure 
a  new  system  and  for  deciding  on  the  most  effective  use  of  limited  resources. 

•  Operational  users  to  support  the  development  of  effective  tactics,  doctrine 
and  procedures.  [Ref.  18:p.  1-1] 

R  TYPES  OF  SHIP  TEST  AND  EVALUATION 

The  principal  types  of  T&E  pertaining  to  ships  are: 

1.  Developmental  Test  and  Evaluation  (DT&E) 

DT&E  is  conducted  to  assist  the  engineering  design  and  development 
process  and  to  verify  attainment  of  technical  performance  specifications  and 
objectives  [Ref.  ll:p.  3]. 

2.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  testing  conducted  on  production  items  to  ensure  systems  meet 
contract  specifications  and  requirements,  usually  for  contractual  acceptance 
purposes.  It  is  a  type  of  DT&E  [Ref.  1  l:p.  5]. 

3.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  conducted  to  determine  a  system’s  operational  effectiveness  and 
operational  suitability,  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  operational  thresholds,  and  develop  tactics  [Ref. 
1  l:p.  5A].  OT&E  has  three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment. 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 

•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 
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4.  The  Dinerence  between  DT&E  and  OT&E 

DT&E  is  focused  on  a  system  meeting  technical  and  production 
specifications,  while  in  OT&E,  the  focus  is  on  assessing  the  actual  functioning  of  the 
system  in  the  realistic  combat  environment,  against  the  user  requirement.  Figure  1 
illustrates  this  principal  difference  between  DT&E  and  OT&E. 


Figure  1:  DT&E  and  OT&E  Comparison 


DT&E  is  planned  and  monitored  by  the  developing  agency  and  is  normally 
initiated  by  the  contractor.  It  includes  the  T&E  of  components,  sub-systems  and 
hardware  and  software  integration,  as  well  as  preproduction  and  production 
qualification  testing.  OT&E,  on  the  other  hand,  is  conducted  by  an  organisation  that 
is  independent  of  the  developer,  on  the  complete  system  in  as  operationally  realistic 
an  environment  as  possible,  with  hostile  forces  representative  of  the  anticipated 
threat  and  with  typical  users  operating  and  maintaining  the  system.  Table  1 
highlights  the  differences  between  DT&E  and  OT&E. 


11 


TABLE  1:  DIFFERENCES  BETWEEN  DT&E  AND  OT&E  [Ref.  18:p,  3-5] 


DT&E 

OT&E 

Controlled  by  Program  Manager 

Controlled  by  Independent  Agency 

One-on-One  Tests 

Many-on-Many  Tests 

Sterile  Controlled  Environment 

Tactical  Environment  with  Operational 
Scenario 

Contractor  Involvement 

No  Contractor  Involvement 

Trained  Experienced  Operators 

Users  Recently  Trained  on  System 

Specific  Performance  Measurements 
and  Goals 

Operational  Effectiveness  and  Operational 
Suitability  Performance  Measurements 

To  further  highlight  the  difference  between  DT&E  and  OT&E,  this 
anecdote,  originally  from  the  Air  Force  Manual  55-43,  1979  and  reprinted  in  the 
U.S.  Defence  Systems  Management  College  Test  and  Evaluation  Guide  [Ref.  I8:p. 
3-3]  is  an  account  of  what  is  probably  the  first  operational  test  and  evaluation; 

The  test  and  evaluation  of  aircraft  started  with  the  contract  awarded  to  the 
Wright  brothers  in  1908.  This  contract  specified  a  craft  which  would  lift  two 
men  with  a  total  weight  of  350  pounds,  carry  enough  fuel  for  a  flight  of  125 
miles,  and  fly  40  miles  per  hour  in  still  air.  The  contract  also  required  that 
testing  be  conducted  to  assure  this  capability.  What  we  now  call  DT&E  was 
satisfied  when  the  Wright  brothers  (the  developer)  demonstrated  that  their 
airplane  could  meet  those  first  contract  specifications.  However,  no  immediate 
military  mission  had  been  conceived  for  the  Wright  Flyer.  It  was  shipped  to 
Fort  Sam  Housten,  Texas  where  Captain  Benjamin  D.  Foulois,  the  pilot,  had 
orders  to  “teach  himself  to  fly”.  He  had  to  determine  the  airplane’s 
performance,  how  to  maintain  it,  and  the  kind  of  organisation  that  would  use 
it.  In  the  process.  Captain  Foulois  subjected  the  Wright  Flyer  to  test  and 
evaluation  in  operational  conditions.  Foulois  soon  discovered  operational 
deficiencies.  For  example,  there  was  no  seat  on  the  airplane.  During  hard 
landings,  Foulois’  130  pound  frame  usually  parted  company  from  the  airplane. 


To  correct  the  problem,  Foulois  bolted  an  iron  tractor  seat  to  the  airplane.  The 
seat  helped  but  Foulois  still  toppled  from  his  perch  on  occasion.  As  a  further 
improvement,  Foulois  looped  his  Sam  Browne  belt  through  the  seat  and 
strapped  himself  in.  Ever  since  then,  contoured  seats  and  safety  belts  -  a 
product  of  this  earliest  “operational”  test  and  evaluation  -  have  been  part  of  the 
military  airplane. 

5.  Operational  Effectiveness  and  Operational  Suitability 

A  system’s  operational  effectiveness  and  operational  suitability  are  the 
fundamental  objects  of  OT&E.  The  USN  and  RAN  definitions  (which  are  possibly 
derived  from  the  USN)  for  these  terms  are  similar. 

a.  Operational  Effectiveness 

(1)  USN  Definition.  (Dperational  effectiveness  is  the  capability  of  the 
system  to  perform  its  intended  function  effectively  over  the  expected  range  of 
operational  circumstances,  in  the  expected  environment,  and  in  the  face  of  the 
expected  threat,  including  countermeasures  where  appropriate  [Ref.  1 1  :p.  8]. 

(2)  RAN  Definition.  Operational  Effectiveness  is  the  capability  of  the 
system  to  perform  its  intended  function  to  the  required  standard,  over  the 
expected  range  of  operational  circumstances,  in  the  expected  environment,  and  in 
the  face  of  the  expected  threat  including  countermeasures  [Ref.  44;p.  A-1]. 

A  subtle  (Ufference  between  these  definitions  is  in  the  description  of  the 
assessed  standard.  The  USN  uses  the  term  “effectively”,  meaning  to  do  the  job 
regardless  of  meeting  specification,  whereas  the  RAN  uses  the  term  “to  the  required 
standard”,  meaning  to  meet  the  specification. 

b.  Operational  Suitability 

(1)  USN  Definition.  Operational  suitability  is  the  capability  of  the 
system,  when  operated  and  maintained  by  typical  fleet  personnel  in  the  expected 


13 


numbers  and  of  the  expected  experience  level,  to  be  reliable,  maintainable, 
operationally  available,  logistically  supportable  when  deployed,  compatible, 
interoperable,  and  safe[Ref.  ll:p.  9]. 


(2)  RAN  Definition.  Operational  suitability  is  the  capability  of  the 
system,  when  operated  and  maintained  by  operational  personnel  in  the  expected 
numbers  and  of  the  expected  experience  level,  to  be  reliable,  maintainable, 
operationally  available  and  logistically  su{^rtable  [Ref.  44:p.  A-2]  in  the  specified 
environment  within  a  specified  time  period.[Ref.  38:p.  14A-3] 

The  relationship  of  operational  effectiveness  and  operational  suitability 
to  OT&E,  and  the  major  questions  addressed  in  OT&E  are  shown  in  Figure  2. 


Figure  2:  Questions  Addressed  in  OT&E  [Ref.  18:p.  3-6] 
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C  AUSTRALIA’S  DEFENCE  ENVIRONMENT 

The  recently  publicly  released  “Australia’s  Strategic  Planning  in  the  1990’s” 
[Ref.  5]  has  not  identified  a  specific  military  threat  to  Australia  in  the  foreseeable 
future.  It  does,  however,  identify  significant  uncertainties  concerning  the  future 
shape  of  the  Australian  strategic  environment. 

1.  Strategy 

Australia’s  national  strategic  policy  has  evolved  over  recent  decades.  It  has 
come  from  a  position  of  defence  dependence  on  allies  through  concentration  on  the 
immediate  needs  of  self  defence  to  a  positive  acceptance  of  both  self-reliance  and 
regional  influence  [Ref.  5:p.  3].  The  latest  Australian  Defence  Corporate  Plan  [Ref. 
8]  highlights  three  central  elements  in  Australia' s  defence  policy; 

•  Seif  Reliance 

•  Regional  Cooperation 

•  Strong  Alliances 

Seif  reliance  accords  priorities  to  meeting  credible  levels  of  threat  in 
Australia’s  area  of  direct  military  interest  [Ref.  6:p.  vii].  Regional  cooperation 
ensures  strategic  stability  and  security  in  the  region,  while  strong  alliances  express 
common  security  interests  and  provide  avenues  for  defence  access  and  strategic 
influence.  Close  defence  relations  with  the  U.S.  remain  central  to  the  policy  of 
defence  self-reliance.  Although  Australia  does  not  depend  on  the  U.S.  for  protection, 
it  derives  significant  benefits  from  close  collaboration  through  access  to  advanced 
technology,  training  and  exercises,  and  intelligence.  These  benefits  give  Australia  a 
level  of  capability  that  would  be  difficult  and  more  costly  for  it  to  achieve  alone. 
[Ref.  8:p.  p  5] 
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2.  Australian  Defence  Budget  and  Policies 

To  achieve  its  defence  goals,  Australia’s  defence  budget  for  1992-93  is  $A 
9.8  billion.  This  accounts  for  about  9%  of  Federal  outlays  and  is  equivalent  to  2.4% 
of  the  expected  GDP.  [Ref.  8:p.  23]  Australia  is  placing  emphasis  on  capital 
investment,  devoting  27.3%  of  its  defence  budget.  It  intends  to  maintain  capital 
investment  at  about  $A  2.5  billion  per  year,  ctf*  which  about  $A  2  billion  is  on  capital 
equipment.  Along  with  this  significant  investment  in  new  capabilities,  Australia  is 
now  buying  far  less  “off  the  shelf’  from  overseas,  with  systems  increasingly  being 
designed  and  built  in  Australia,  or  adapted  from  overseas  designs.  [Ref.  8:p.  17]  As 
a  result,  Australian  is  now  building  frigates  and  submarines  due  to  enter  service  in 
the  mid  1990’s. 

The  Australian  Government’s  forward  estimates  indicate  that  the  Defence 
Budget  will  be  reduced  by  0.5%  in  19^-94  and  then  maintained  at  this  level  in  real 
terms  in  1994-95  and  199S%  [Ref.  8:p.  24].  Over  the  four  years  to  1996,  reductions 
of  up  to  6,600  permanent  military  personnel  (about  10%  of  total  force)  and  1785 
Defence  civilians  (about  8%)  are  anticipated  [Ref.  8:p.  14]. 

The  area  over  which  the  Australian  Defence  Force  (ADF)  needs  to  operate, 
the  importance  of  denying  an  adversary  freedom  in  the  sea  and  air  approaches, 
Australia’s  limited  population  base  and  the  characteristics  of  the  northern  Australian 
area,  place  a  premium  on  technologically  advanced  systems  rather  than  manpower 
intensive  forces.  As  the  costs  of  local  development  and  production  of 
technologically  advanced  systems  are  high,  development  emphasis  is  placed  on 
those  areas  unique  to  Australia  or  where  particular  skills  have  ongoing  relevance  to 
a  range  of  defence  capabilities. [Ref.  5;pp.  28, 38] 

In  summary,  Australia  is  contributing  to  its  strategic  goals  and  ensuring  that 
the  ADF  remains  a  formidable  military  force  in  regional  terms,  by  maintaining 
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priority  on  capital  investment.  The  focus  is  towards  a  smaller  and  more  technology 
oriented  force,  with  a  more  streamlined  and  effective  organisation.  Therefore, 
Australia  needs  to  ensure  that  its  more  technology  advanced  systems  are 
operationally  effective  and  operationally  suitable  for  the  missions  required  of  them. 

3.  U.S.  Versus  Australia 

Of  a  similar  physical  size  but  much  larger  population,  the  U.S.  is  an 
economic  and  military  superpower  compared  to  Australia.  Table  2  compares  some 
significant  characteristics  of  the  two  nations. 


TABLE  2:  U.S.  VS.  AUSTRALIA  1991/92  [Ref.  10] 


1991  Statistics 

U.S. 

Australia 

Area 

3,021,295  sq  m 

2.966,151  sq  m 

Population 

251.8  million 

17. 1  million 

GDP 

$US  5,673.9  billion 

$A  376.3  billion 
($US  293.2  billion) 

Growth 

-0.8% 

-2.2% 

Inflation 

43% 

3.3% 

Debt 

$US  828.5  billion 

$US  125.0  billion 

Defence  Budget 

$US  290.9  billion 

$A  9.44  billion 
($US  7.27  billion) 

%  of  Federal  Outlay 

22% 

9% 

Total  Armed  Forces 

1,913,750 

67,900 

Spending  on  Capital 

$US  71.7  billion 

$A  1.92  billion 

Equipment 

($US  1.40  billion) 

%  of  Defence  Budget 

26.4% 

20.4% 

R  &  D  Expenditure 

$US  36.2  billion 

$A  0.22  billion 
($US  0. 16  billion) 

%  of  Defence  Budget 

12.4% 

2.3%  j 
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a.  Differences 

The  “National  Military  Strategy  of  the  United  States”  [Ref.  7]  affirms 
that  U.S.  Defence  policy  is  built  upon  four  key  foundations; 

(1)  Strategic  Deterrence  and  Defence.  The  significant  political 
instabilities  in  the  former  Soviet  Union  and  the  threat  posed  by  the  increasing 
numbers  of  potentially  hostile  states  developing  weapons  of  mass  destruction 
requires  that  the  maintenance  of  a  modern,  fully  capable  and  reliable  strategic 
deterrent  remains  the  first  defence  priority. 

(2)  Forward  Presence.  The  day  to  day  presence  of  U.S.  forces  in 
regions  vital  to  U.S.  national  interests  is  key  to  averting  crises  and  preventing  war. 
In  addition  to  forces  based  overseas  and  afloat,  forward  presence  includes  periodic 
and  rotational  deployments,  access  and  storage  agreements,  combined  exercises, 
security  and  humanitarian  assistance,  port  visits  and  military  to  military  contacts. 

(3)  Crisis  Response.  The  capability  to  respond  to  regional  crises  on 
short  notice  is  a  key  demand  of  U.S.  strategy.  This  response  may  range  from  a  single 
discriminate  strike  to  the  employment  of  overwhelming  force  to  defeat  a  regional 
aggressor. 

(4)  Force  Reconstitution.  Reconstitution  is  required  to  preserve  a 
credible  capability  to  forestall  any  potential  adversary  from  competing  militarily 
with  the  U.S.  It  involves  forming,  training  and  fielding  new  fighting  units  if  required 
at  short  notice.  It  also  involves  maintaining  technology,  doctrine,  training, 
experienced  military  personnel  and  innovation  necessary  to  retain  the  competitive 
edge  in  decisive  areas  of  potential  military  competition. 

This  strategy  is  founded  on  the  premise  that  the  United  States  will 
provide  the  leadership  needed  to  promote  global  peace  and  security  [Ref.  7:p.  6].  It 
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does,  however,  reflect  the  shift  from  containing  the  spread  of  communism  and 
deterring  Soviet  aggression  to  a  more  diverse,  flexible  strategy  which  is  regionally 
oriented  and  capable  of  responding  decisively  to  future  challenges  [Ref.  7:p.  1]. 
Although  the  U.S.  and  Australian  strategic  policy  elements  may  not  be  directly 
comparable,  the  U.S.  places  a  greater  emphasis  on  Defence  with  strategies  of  world 
leadership,  global  influence  and  self  sufficiency,  while  Australia  places  emphasis  on 
regional  influence  and  self-reliance. 

To  achieve  these  goals,  the  U.S.  devotes  a  considerably  higher 
proportion  of  its  Federal  expenditure  to  E>efence,  than  does  Australia.  It  also  spends 
a  considerably  greater  percentage  of  its  Defence  Budget  on  R&D.  U.S.  policy  is  to 
maintain  worid  leadership  in  key  Defence  technologies,  by  continuously  improving 
the  process  of  indentifying  and  introducing  new  technologies  [Ref.  9].  Although 
R&D  is  conducted  with  emphasis  on  those  areas  unique  to  Australia,  lower  risk, 
proven  technologies  are  preferred  for  the  majority  of  new  Australian  Defence 
acquisitions. 

h.  Similarities 

Like  Australia,  the  U.S.  current  projections  for  FY  92-95  show  a 
continued  reduction  of  the  Defence  Budget  by  about  2%  per  annum.  Additionally, 
with  the  recent  change  to  a  Democratic  Administration  under  President  Clinton, 
further  reductions  in  Defence  expenditure  can  be  expected.  During  this  period  of 
budget  and  trade  deficits  and  urgent  domestic  needs,  the  U.S.  military  strategy  will 
be  implemented  within  a  significantly  reduced  Defence  budget,  and  so  places  a 
premium  on  efficiency  without  compromising  effectiveness  [Ref.  7:p.  4]. 

As  does  Australia,  the  U.S.  relies  on  technological  superiority  to  offset 
quantitative  disadvantages,  to  minimise  risk  to  its  forces,  and  to  enhance  deterrence 
and  the  potential  for  swift,  decisive  termination  of  conflict  [Ref.  7:p.  10].  The  U.S 
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also  realises  it  can  meet  the  challenges  of  the  foreseeable  future  with  a  much  smaller 
force  than  it  has  had  in  recent  years  [Ref.  7],  and  that  implications  of  a  25%  force 
cut  by  1995  may  be  possible  [Ref.  10].  Despite  these  funding  and  program  cutbacks, 
the  U.S.  is  also  building  new  surface  combatants  and  submarines.  The  first  DDG-51 
“Arleigh  Burke”  completed  testing  and  the  first  SSN-21  “Seawolf’  is  under 
construction. 

A  significant  similarity  between  the  U.S.  and  Australia  is  one  of 
culture.  Both  nations  have  predominantly  Caucasian,  English  speaking  populations, 
a  democratic  system  of  government,  close  economic,  military  and  political  ties,  and 
share  similar  cultural  values  and  ideology. 

Despite  being  significantly  larger  than  Australia  in  economic  and 
military  terms,  and,  consequently,  having  a  different  Defence  strategy,  the  U.S.  is 
similar  to  Australia  in  many  areas.  These  include  reducing  budgets  and  size  of  the 
Defence  force,  a  requirement  for  advanced  technology,  and  a  need  for  efficiency 
while  maintaining  effectiveness.  But  perhaps  above  all,  a  similar  culture  which 
promotes  a  relative  ease  of  communication  and  assimilation  of  ideas,  concepts, 
procedures  and  methodologies  between  the  two  nations. 

4.  United  States  Navy  Versus  Royal  Australian  Navy 

A  similar  contrast  applies  to  the  navies  of  both  nations.  As  Table  3  shows, 
the  USN  is  far  larger  and  more  capable  than  the  RAN.  However,  there  are  number 
of  similarities.  Although  the  RAN  had  its  origins  in  the  Royal  Navy  and  with  British 
systems,  since  the  1960’s  the  greater  influence  on  the  RAN  has  been  from  the  USN. 
The  purchase  of  three  modified  Charles  E  Adams  (DDG-2)  class  destroyers  from  the 
U.S.  in  the  mid  1960’s  heralded  an  increasingly  close  relationship  between  the  two 
navies.  The  purchase  of  four  Oliver  Hazard  Perry  (FFG-7)  class  frigates  from  the 
U.S.  in  the  early  1980’s,  and  the  local  construction  of  two  more  of  the  class  in 
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Australia,  continued  this  relationship.  Although  the  RAN’s  needs  no  longer  support 
the  purchase  of  current  U.S.  ship  types,  many  new  systems  and  equipment  are  still 
purchased  through  the  USN.  Ongoing  support  for  these  U.S.  ships  currently  in  RAN 
service,  and  Australia’s  emphasis  on  interoperability  with  allies,  ensure  a  continuing 
close  association.  The  RAN  shares  with  the  USN  much  common  equipment  and 
weapons,  compatible  operational  procedures,  similar  logistic  support  systems  and 
defence  acquisition  procedures. 


TABLE  3:  USN  VS.  RAN  1991  [Ref.  10] 


USN 

RAN 

Personnel 

546,650 

15,300 

Submarines 

110 

5 

Aircraft  Carriers 

14 

- 

Cruisers 

48 

- 

E)estroyers 

45 

3 

Frigates 

83 

8 

Patrol  (inc.  Coast  Guard) 

30 

15 

Mine  Warfare 

24 

2 

Amphibious 

65 

1 

Support  &  Misc. 

162 

12 

Total  Ships: 

581 

46 

Major  Ship 

Acquisition  Programs 

DDG-51  class  destroyers 
SSN-21  class  submarines 

ANZAC  class  frigates 
COLLINS  class  subs. 

D.  OT&E  BACKGROUND  SUMMARY 

OT&E  forms  a  fundamental  part  of  minimising  risk  in  the  weapons  system 
acquisition  process  and  in  future  system  operations.  It  is  conceptually  different  from 
DT&E,  being  focussed  on  assessing  the  operational  effectiveness  and  operational 
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suitability  of  the  system  in  a  realistic  combat  environment,  with  respect  to  the  user 
requirement. 

Australia  is  placing  emphasis  on  capital  investment  for  Defence,  particularly  for 
the  RAN,  with  new  frigates  and  submarines  presently  under  construction.  In 
response  to  budget  reductions  the  current  focus  of  the  RAN  is  towards  a  smaller  and 
more  technology  oriented  force,  with  a  more  streamlined  and  effective  organisation. 

The  U.S.  and  Australia  have  a  number  of  similarities,  perhaps  the  most 
important,  a  similar  culture  which  promotes  a  relative  ease  of  communication  and 
assimilation  of  ideas,  concepts,  procedures  and  methodologies  between  the  two 
nations.  The  USN  is  a  world  leader  in  the  development  of  high  technology  weapons 
systems  and  has  extensive  experience  in  their  acquisition,  testing  and  operation. 

With  new  frigates  and  destroyers  due  to  enter  service  in  the  mid  1990s,  the  RAN 
has  an  urgent  need  for  an  efficient,  effective  OT&E  system  to  ensure  the  operational 
risk  of  these  new  combatants  is  minimised,  and  the  combatants  meet  the  current 
requirements  of  the  RAN. 
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III.  USN  SHIP  OPERATIONAL  TEST  AND  EVALUATION 

This  chapter  describes  the  USN  OT&E  system.  It  explains  the  OT&E 
relationship  and  importance  to  the  U.S.  Department  of  Defence  (DoD)  acquisition 
process,  the  specific  OT&E  policies  and  methods  employed  in  ship  OT&E  and  why 
the  OT&E  of  ships  is  different  from  other  systems.  The  USN  system  of  ship 
acceptance  and  the  input  of  OT&E  into  the  process  is  also  described.  The  OT&E  of 
the  DDG-51  class  destroyer,  the  latest  class  of  warship  to  undergo  OT&E.  is  used  as 
an  example  of  the  implementation  of  USN  OT&E  policies.  The  workings  of  the 
USN  system  are  well  documented  and  the  OT&E  system  can  be  traced  from  the 
statute  requirements  of  Congress,  through  DoD  directives  to  implementation  by 
Navy  specific  instructions. 

A.  U.S.  DEPARTMENT  OF  DEFENCE  ACQUISITION  PROCESS 

The  USN  OT&E  system  forms  an  integral  part  of  the  US  DoD  acquisition 
process.  In  1987,  the  Defence  system  acquisition  process  underwent  revision  in  an 
attempt  to  make  it  less  costly,  less  time  consuming,  and  more  responsive  to  the  needs 
of  the  operational  community.  The  Defence  system  life  cycle  consists  of  five  phases 
of  the  acquisition  process: 

•  Concept  Exploration/Definition 

•  DemonstrationA^alidation 

•  Engineering  and  Manufacturing  Development 

•  Full  Rate  Production/Deployment 

•  Operational  Support 
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As  Figure  3  shows,  these  phases  are  separated  by  major  decision  points 
(milestones)  when  a  program  is  reviewed  and  authorised  to  advance  to  the  next  stage 
in  the  cycle,  that  provide  a  basis  for  progressive  decisionmaking  associated  with 
program  maturation.  T&E  results  and  planned  T&E  play  an  important  part  and  are 
rigorously  assessed  as  part  of  the  milestone  reviews  [Ref.  18:p.  1-3].  These 
milestones  are: 

•  Milestone  O  -  Concept  Studies  Approval 

•  Milestone  1  -  Concept  Demonstration  Approval 

•  Milestone  II  -  Development  Approval 

•  Milestone  IIA  -  Low  Rate  Initial  Production 

•  Milestone  III  -  Full  Rate  Production 

•  Milestone  IV  -  Major  Modification  Approval  (as  required) 
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R  TYPES  OF  USN  SHIP  TEST  AND  EVALUATION 

The  USN  Instruction  on  Test  and  Evaluation  [Ref.  11]  defines  the  principal 
types  of  T&E  pertaining  to  ships  as  follows; 

1.  Developmental  Test  and  Evaluation  (DT&E) 

DT&E  is  conducted  to  assist  the  engineering  design  and  development 
process  and  to  verify  attainment  of  technical  performance  specifications  and 
objectives  [Ref.  1  l:p.  3].  DT&E  is  conducted  in  three  major  phases. 

2.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  testing  conducted  on  production  items  to  ensure  systems  meet 
contract  specifications  and  requirements,  and  is  a  type  of  DT&E  [Ref.  1  l:p.  5].  The 
USN  regard  trials  of  new  ships  conducted  by  the  Board  of  Inspection  and  Survey 
(INSURV)  and  shipyard  industrial  testing  at  PAT&E  [Ref.  32:p.  2-5].  Two  phases  of 
ship  PAT&E  are  defined: 

•  Ship  Construction  Tests  and  Trials  consist  of  all  testing  conducted  on  the  ship 
during  construction,  including  INSURV’s  Acceptance  Trials. 

•  Ship  Post-Delivery  Tests  and  Trials  are  the  conventional  tests  and  trials, 
including  INSURV’s  Final  Contract  Trials,  that  commence  after  ship 
delivery  and  continue  to  the  end  of  the  SCN  obligation  or  woric  limiting  date. 

3.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  conducted  to  determine  a  system’s  operational  effectiveness  and 
operational  suitability,  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  OT  thresholds,  and  develop  tactics  [Ref.  1  l;p. 
5A].  OT&E  has  three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment. 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 
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•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 

OT&E  is  subdivided  into  two  major  categories: 

•  Initial  OT&E  (lOT&E)  (OT-I  and  OT-II)  which  is  up  to  and  including 
OPEVAL  (final  phase  of  OT-2). 

•  Follow-on  OT&E  (FOT&E)  (OT-III  and  OT-IV)  which  is  all  OT&E  after  the 
final  phase  of  OPEVAL. 

C.  OT&E  IN  THE  ACQUISITION  PROCESS 

According  to  the  Honorable  Gerald  A.  Cann,  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition: 

Test  and  evaluation  is  the  major  control  mechanism  in  the  acquisition  process 
for  assessing  system  performance  against  system  requirements.  Navy  decision 
makers  rely  on  solid  testing  results,  both  developmental  and  operational,  to 
provide  the  analytical  groundwork  for  forming  judicious  program 
decisions.  [Ref.  12] 

This  statement  by  the  USN’s  senior  acquisition  executive,  highlights  the 
importance  of  T&E  within  the  USN.  Program  advance  from  one  phase  to  the  next  is 
not  by  the  calendar  of  planned  schedule,  but  by  actual  resolution  of  critical 
operational  issues  and  achievement  of  pre-set  thresholds  verified  by  T&E  [Ref. 
1  l:p.  2].  The  importance  of  OT&E,  in  particular,  to  the  progressive  assessment  of 
programs  is  such  that  its  use  has  been  mandated  by  the  U.S.  Congress  and 
incorporated  into  the  laws  of  the  United  States. 

1.  Statute  Requirement 

The  requirement  for  adequate  OT&E  of  Defence  programs  has  strong 
congressional  support  and  is  mandated  by  law.  The  law  includes  the  provision  of  a 
Director  of  Operational  Test  and  Evaluation  in  the  Department  of  Defence, 
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appointed  from  civilian  life  by  the  President,  by  and  with  the  advice  and  consent  of 
the  Senate  [Ref.  16].  The  law  also  addresses  specific  areas  of  OT&E  reporting  and 
conduct  to  ensure  the  Congress  is  kept  informed,  and  the  testing  and  reporting  are 
impartial  [Ref.  17].  By  including  the  requirement  that  a  major  Defence  acquisition 
program  may  not  proceed  beyond  low-rate  initial  production  (i.e.  for  Milestone  III 
-  Full  Rate  Production  Decision)  until  Initial  OT&E  of  the  program  is  completed. 
Congress  ensures  that  OT&E  is  an  integral  part  of  the  acquisition  process. 

2.  OT&E  Contributions  at  Major  Milestones 

T&E  progress  is  monitored  by  the  Office  of  the  Secretary  of  Defence 
(OSD)  throughout  the  acquisition  process.  For  major  Defence  acquisition  programs, 
the  Director,  T&E  and  the  Director,  OT&E  within  OSD  render  independent 
assessments  to  the  Defence  Acquisition  Board,  the  Defence  Acquisition  Executive 
and  Secretary  of  Defence  at  each  major  milestone.  These  T&E  officials  also  assess 
the  T&E  results  for  less  than  major  defence  acquisition  programs  that  are 
specifically  designated  by  OSD  as  OSD  T&E  Oversight  Programs.  The  assessments 
are  based  on  the  following  T&E  information:  [Ref.  18:p.  2-5] 

•  The  Test  &  Evaluation  Master  Plan  (TEMP)  and  more  detailed  supporting 
documents  developed  by  responsible  Service  activities. 

•  Service  operational  test  agency  reports  and  briefings. 

•  Developmental  test  and  evaluation  data. 

a.  OT&E  prior  to  Milestone  / 

During  the  Concept  Exploration  /  Definition  Phase  prior  to  Milestone  I, 
laboratory  testing,  modeling  and  simulations  are  conducted  to  demonstrate  and 
assess  the  capabilities  of  key  subsystems  and  components.  The  test  and  simulation 
designs  are  based  on  the  requirements  documented  in  the  Mission  Need  Statement 
(MNS).  The  Operational  Test  and  Evaluation  Agency  (OTA)  monitors  concept 
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exploration  T&E  to  gather  information  for  future  test  and  evaluation  planning  and 
to  provide  effectiveness  and  suitability  inputs  desired  by  the  Program  Manager.  The 
OTA  also  conducts  operational  assessments,  as  feasible,  to  assess  the  operational 
impact  of  candidate  technical  approaches  and  to  assist  in  selecting  preferred 
alternative  systems  concepts.  [Ref.  18;p.  2-5] 

b.  OT &E  prior  to  Milestone  II 

During  the  Demonstration/Validation  phase  prior  to  Milestone  II, 
concepts  approved  for  demonstration  and  validation  form  the  baseline  that  is  used 
for  detailed  test  planning.  The  OTA  conducts  early  operational  assessments  to 
identify  the  best  approach,  indicate  the  risks  and  solutions  for  this  phase  of 
development,  examine  operational  aspects  of  the  systems  development,  and 
estimate  potential  operational  effectiveness  and  suitability.  This  OT&E  phase  is  an 
assessment  only,  with  no  actual  testing  being  conducted.  Typical  operational  and 
support  personnel  are  used  to  obtain  a  valid  estimate  of  the  user’s  capability  to 
operate  and  maintain  the  system.  The  user  ol  the  system  monitors  test  and  evaluation 
during  the  concept  E)emonstration  and  Validation  phase.  Among  the  most  important 
products  of  user  monitoring  are  the  attainment  of  early  orientation  and  training, 
demonstrations  of  system  performance  and  valid  operational  test  assessments  of 
systems  maintainability  and  supportability.[Ref.  18;p.  2-7] 

c.  OT&E  prior  to  Milestone  III 

The  objective  of  the  Engineering  and  Manufacturing  Development 
(EMD)  phase  prior  to  Milestone  III,  is  to  design,  fabricate  and  test  a  preproduction 
system  that  closely  approximates  the  final  product.  T&E  activities  intensify  during 
EMD,  culminating  in  OPEVAL,  and  make  significant  contributions  to  the  overall 
acquisition  decision  process.  [Ref.  18:p.  2-9]  OT&E  conducted  prior  to  the 
production  decision  is  for  the  following  reasons: 
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•  Estimate  the  operational  effectiveness  and  suitability  of  the  system. 

•  Identify  operational  deficiencies. 

•  Recommend  and  evaluate  changes  in  production  configuration. 

•  Provide  information  for  developing  and  refining  logistics  support 
requirements  for  the  system  and  training,  tactics,  techniques,  and  doctrine. 

•  Provide  information  to  refine  operation  and  support  cost  estimates,  and  to 
identify  system  characteristics  or  deficiencies  that  can  significantly  impact 
these  costs. 

•  Determine  whether  the  technical  publications  and  support  equipment  are 
adequate. 

•  Estimate  the  survivability  of  the  system  in  the  operational  environment. 

d,  OT&.E  after  the  Production  Decision 

OT&E  activities  continue  after  the  production  decision  in  the  form  of 
Follow-on  Operational  Test  and  Evaluation  (FOT&E)  consisting  of  OT-III  and  OT- 
IV.  It  is  accomplished  to  verify  the  operational  effectiveness  and  suitability  of  the 
production  system  and  to  determine  if  deficiencies  identified  during  initial  OT(&E 
(OT-III)  have  been  corrected.  A  second  phase  of  FOT&E  (OT-IV)  may  be  conduced 
by  the  user  to  refine  doctrine,  tactics,  techniques  and  training  programs  over  the  life 
of  the  system.  [Ref.  18:p.  2-10] 

D.  US  OT&E  ORGANISATION 

Major  participants  in  the  US  OT&E  process  are  listed  in  Table  4. The  US  OT&E 
community  consists  of  a  mix  of  service  and  OSD  offices  responsible  for  planning, 
programming,  budgeting  and  evaluating  operational  tests.  Bgure  4  gives  an  outline 
of  the  OT&E  organisation. 
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TABLE  4:  USN  OT&E  FUNCTIONAL  AUTHORITIES 


OT&E  Function 

USN  Authority 

Policy  Maker  &  Overseer 

Director,  Operational  Test  &  Evaluation  (DoD) 

Sponsor 

Chief  of  Naval  Operations 

Developing  Agency 

Commander.  Naval  Sea  Systems  Command 

Operational  Tester  &  Evaluator 

Commander,  Operational  Test  &  Evaluation  Force 

User 

Reet  Commanders,  Ships’  Commanding  Officers 

Coordinator 

Director,  T&E  and  Technology  Requirements 

The  US  OT&E  community  consists  of  a  mix  of  service  and  OSD  offices 
responsible  for  planning,  programming,  budgeting  and  evaluating  operational  tests. 
Figure  4  gives  an  outline  of  the  OT&E  organisation. 


Figure  4:  U.S.  OT&E  Organisation 
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1.  Policy  Maker  and  Overseer 

The  Director  of  OT«fe  E  (DOT&E)  is  the  principal  OT&E  official  in  the 
Department  of  Defence  and  the  principal  advisor  to  the  Secretary  of  Defence  on 
OT&E.  He  is  responsible  for  policy  formulation,  evaluation  and  oversight  of  OT&E 
within  DoD.  With  status  comparable  to  an  Assistant  Secretary  of  Defence,  DOT&E 
is  independent  of  other  DoD  officials  and  reports  directly  to  the  Secretary  of 
Defence  and  additionally  to  Congress.  His  responsibilities  include  the  following 
[Ref.  13:p.  7-4]: 

•  Monitoring  and  reviewing  all  OT&E  within  DoD. 

•  Designating  observers  to  be  present  during  preparation  for  and  conduct  of  the 
testing  portion  of  OT&E. 

•  Controlling  joint  OT&E  and  coordinating  OT&E  conducted  by  more  than 
one  Military  Department  or  Defence  Agency. 

•  Analysing  the  results  of  major  system  acquisition  OT&E.  For  major  systems 
and  DOT&E  oversight  programs,  reporting  to  SECDEF  and  to  Congressional 
Armed  Services  and  Appropriations  Committees  that  OT&E  is  adequate  and 
confirms  effectiveness  and  suitability  for  combat  systems  tested. 

•  Making  recommendations  to  SECDEF  on  all  budgetary  and  financial  matters 
pertaining  to  OT&E,  including  facilities  and  equipment. 

•  Approving  OT&E  plans  for  major  Defence  acquisition  programs  and 
DOT&E  oversight  programs. 

The  office  of  DOT&E  was  established  because,  although  each  of  the 
Services  have  had  their  own  OT&E  agencies.  Congress  perceived  that  operational 
testing  was  not  as  objective  as  it  should  be.  It  wanted  a  layer  of  oversight  over  the 
procurement  process,  especially  OT&E.  As  a  result,  DOT&E  also  has  reporting 
requirements  to  Congress  as  well  as  the  Secretary  of  Defence.  [Ref.  19] 


31 


2.  Sponsor 

The  Program  Sponsor  is  responsible  for  acquisition  program  requirements 
and  related  system  thresholds.  The  requirements  are  determined  based  on  continuing 
assessments  of  current  and  projected  capabilities  in  the  context  of  changing  military 
capabilities  and  national  Defence  policy.  Within  the  USN,  the  Sponsor  works  for  the 
Chief  of  Naval  Operations  (CNO).  For  major  warship  programs  the  sponsor  is  N8  - 
Warfare  Requirements.  Under  the  new  USN  organisation,  increasing  importance  is 
given  to  the  Fleet  and  Type  Commanders  in  determining  requirements  and 
allocating  priorities.  Besides  generating  the  user  requirement,  the  Sponsor  also  has 
a  review  function  in  the  OT&E  process.  [Ref.  20] 

3.  Developing  Agency 

When  a  program  achieves  Milestone  0,  a  program  office  is  formed  to 
manage  the  acquisition  of  the  system.  For  ships  and  ship  systems,  this  office  may  be 
established  within  the  Naval  Sea  Systems  Command  (NAVSEA)  or  is  affiliated  with 
a  Program  Executive  Officer,  being  ‘double  hatted’,  with  responsibilities  to  both  the 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition),  and  to 
NAVSEA.  The  program  office  is  responsible  for  the  conduct  of  DT«feE  in 
preparation  for  OT&E.  To  assist  all  NAVSEA  programs  with  T&E  matters, 
NAVSEA  has  established  its  own  T&E  office  (SEA  60)  whose  purpose  is  to  [Ref. 
23]: 

•  Assist  Program  Managers  in  complying  with  policies  and  incorporating 
lessons  learned  of  the  past. 

•  Review  T&E  aspects  of  programs  at  major  milestones 

•  Keep  COMNAVSEA  and  PEOs  appraised  of  overall  posture  of  T&E. 

•  Influence  OSD  and  Navy  T&E  policy. 
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Although  primarily  a  DT«&E  policy  authority,  this  office  also  has  an  input 
into  the  OT«&E  process  by  reviewing  the  TEMP,  and  by  conducting  OT&E  readiness 
reviews  of  NAVSEA  programs. 

4.  Operational  Tester  &  Evaluator 

The  Operational  Test  and  Evaluation  Force  (OPTEVFOR)  is  the  USN’s 
sole  independent  operational  test  agency  responsible  for  the  planning  and  conduct 
of  OT&E.  COMOPTEVFOR  reports  directly  to  the  CNO.  and  is  separate  and 
distinct  from  the  developing,  procuring  commands  and  the  user.  Results  of 
OPTEVFOR  evaluations  are  reported  directly  to  CNO  and  to  DOT&E. 
OPTEVFOR’s  mission  is  to  test  and  evaluate  weapons  systems,  ships  aircraft,  and 
equipment  in  the  anticipated  operational  environment  and  against  the  anticipated 
threat:  to  develop  and  validate  procedures  and  tactics  employing  these  weapons 
systems,  ships,  aircraft  and  equipment:  and  when  directed  by  the  CNO,  to  assist 
developing  agencies  in  the  accomplishment  of  DT&E  [Ref.  14:p.  1].  OPTEVFOR 
involvement  in  early  phases  of  research  and  development  includes  inputs  to  the  Test 
and  Evaluation  Master  Plan  (TEMP),  observing  development  testing,  and 
conducting  those  phases  or  operational  testing  necessary  to  provide  CNO  with  an 
early  and  independent  operational  assessment  [Ref.  13:p.  7-5]. 

5.  User 

The  Commanding  Officer  and  crew  of  the  ship  under  test,  and  his  Fleet  and 
Type  Commanders  are  the  ultimate  users  of  the  system.  The  personal  assessment  of 
the  Commanding  Officer  of  the  ship  under  test  is  sought  by  OPTEVFOR  during  the 
conduct  of  OT&E  [Ref.  15:p.  44]. 
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6.  Coordinator 

CNO  has  responsibility  for  ensuring  the  adequacy  of  the  Navy’s  overall  test 
and  evaluation  program.  T«feE  policy  and  guidance  are  exercised  through  the 
Director,  Test,  Evaluation  and  Technology  Requirements  (N091),  This  organisation 
also  acts  as  a  T&E  Focal  Point  and  Coordinator,  responsible  for  coordination  of 
T&E  matters  in  the  designated  Programs,  System  Commands  and  Department  of  the 
Navy,  [Ref.  13:p.  7-5] 

E.  USN  OT&E  POLICY 

USN  OT&E  policy  has  its  basis  in  the  statutes  approved  by  Congress.  This 
policy  is  refined  in  the  DoD  acquisition  Directive  5000.1  [Ref.  25]  and  DoD 
Instruction  5000.2  [Ref.  24].  The  essential  T&E  elements  in  these  directives  are  then 
further  detailed  in  OPNAV  Instruction  3960. IOC  [Ref.  1 1]  and  other  subordinate 
instructions.  This  hierarchy  of  documentation  establishes  the  disciplined 
management  approach  to  OT&E  taken  by  the  USN. 

Basically,  USN  test  and  evaluation  programs  are  structured  to  [Ref.  24:p.  8-2]. 

•  Provide  essential  information  for  assessment  of  acquisition  risk  and  for 
decision  making. 

•  Verify  attainment  of  technical  performance  specifications  and  objectives. 

•  Verify  that  systems  are  operationally  effective  and  suitable  for  intended  use. 

•  Provide  essential  information  in  support  of  decision  making. 

OT&E  programs,  in  particular,  are  structured  to  determine  the  operational 
effectiveness  and  suitability  of  a  system  under  realistic  combat  conditions  and  to 
determine  if  the  minimum  acceptable  operational  performance  requirements  as 
specified  in  the  Operational  Requirements  Document  have  been  satisfied.  U.S. 
OT&E  policy  requires[Ref.  24:p.  8-5]: 
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•  Threat  representative  forces  be  used  whenever  possible. 

•  Typical  users  to  operate  and  maintain  the  system  under  conditions  simulating 
both  combat  stress  and  peacetime  conditions. 

•  Production  or  production  representative  articles  be  used  for  the  dedicated 
phase  of  OT&E  that  supports  the  full  rate  production  decision. 

The  ultimate  uses  of  information  obtained  through  OT&E  are  feedback  to  the 
Developing  Agency  on  the  operational  strengths  and  weaknesses  of  the  system,  and 
inputs  for  decisions  to  proceed  with  production  and  fleet  introductions.  A  number  of 
key  factors  of  note  are  highlighted  in  USN  OT&E  policy: 

1.  Independence 

The  fundamental  EtoD  Directive  on  Etefence  acquisition  [Ref.  25:p.  1-8] 
requires  an  independent  operational  test  activity.  Each  Military  Department  is 
required  to  establish  an  independent  operational  test  and  evaluation  activity.  This 
activity  is  required  to: 

•  Be  separate  and  independent  from  the  materiel-developing  and  procuring 
agency  and  the  using  agency. 

•  Be  responsible  for  planning  and  conducting  operational  tests,  reporting 
results,  and  providing  evaluations  of  each  tested  system’s  operational 
effectiveness  and  suitability. 

•  Report  directly  to  the  head  of  the  DoD  Component,  except  that  the  Secretary 
of  a  Military  Department  may  delegate  responsibility  for  supervising  this 
activity  to  the  Service  Chief  concerned. 

Acquisition  managers  are  not  to  influence  or  attempt  to  influence  the 
objectivity  and  completeness  of  test  results  presented  to  decisionmakers  by  the 
independent  operational  test  activity.  Independence  is  necessary  so  that  there  is  no 
question  of  impropriety.  The  testing  and  reporting  agency  has  nothing  to  gain  or  lose 
by  whatever  they  say.  As  major  acquisition  decisions  hinge  on  the  results  of  OT&E, 
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independence  is  necessary  to  ensure  that  the  reports  on  which  those  decisions  are 
based  are  not  self  serving.  [Ref.  19] 

2.  Use  of  Contractors 

The  use  of  system  contractors  in  support  of  OT&E  conducted  to  support  a 
decision  to  proceed  beyond  low  rate  Initial  production  (Milestone  III)  is  restricted 
by  law  [Ref.  17]  to  ensure  impartiality  of  testing.  No  person  employed  by  the 
contractor  for  the  system  being  tested  may  be  involved  in  OT&E,  unless  contractors 
are  planned  to  be  involved  when  the  system  is  deployed  in  combat. 

3.  T&E  Planning 

Test  and  evaluation  planning  is  considered  by  the  USN  to  be  fundamental 
to  the  conduct  of  OT&E.  The  Test  and  Evaluation  Master  Plan  (TEMP)  is  the 
controlling  T&E  document  and  is  considered  by  some  to  be  the  single  most 
important  document  associated  with  an  acquisition  program.  The  TEMP  is  directive 
in  nature  and  defines  and  integrates  test  objectives,  critical  issues,  system 
characteristics,  test  responsibilities  resource  requirements  and  test  schedules.  An 
approved  TEMP  constitutes  direction  to  conduct  the  specified  T&E.[Ref.  15:p.  6-1] 

Drafted  by  the  Program  Manager  (PM),  the  TEMP’s  main  purpose  is  to 
combine  the  PM’s  DT&E  and  COMOPTEVFOR’s  OT&E  into  one  integrated 
master  plan.  [Ref,  ll:p.  11]  OPTEVFOR  draft  Part  Four  on  OT&E  and  provide 
OT&E  resource  requirements  for  Part  Five.  The  TEMP  is  then  co-submitted  by  both 
the  Program  Manager  and  COMOPTEVFOR.  For  ship  programs,  the  TEMP  is 
reviewed  by  the  Assistant  Chief  of  Naval  Operations  (Surface  Warfare),  the 
Director,  T&E  and  Technology  Requirements,  and  the  Assistant  Secretary  of  the 
Navy  (Research,  Development  and  Acquisition).  The  TEMP  is  ultimately  approved 
at  the  DoD  level  by  the  Director,  OT&E  and  the  Under  Secretary  of  Defence  for 
Acquisition,  Director  Test  and  Evaluation. 
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Approval  of  the  TEMP  constitutes  establishment  of  a  contract  between  the 
requirement-setter  and  the  PM  which  is  essential  to  formal  and  accountable  program 
execution,  and  between  the  requirement-setter  and  the  OT  agency,  upon  which 
COMOPTEVFOR  will  independently  evaluate  system  operational  effectiveness 
and  suitability.  An  approved  TEMP  is  required  at  Milestone  I  and  is  updated  for  each 
subsequent  milestone  and  when  significant  program  changes  occur.  The  TEMP  also 
serves  several  secondary  purposes: 

•  It  allows  all  involved  to  see  exactly  what  hurdles  the  system  must  clear. 

•  It  allows  the  PM  to  make  good  projections  of  COMOPTEVFOR’s  OT&E 
costs,  which  must  be  funded. 

•  It  allows  fleet,  range,  simulator  and  target  schedulers  to  plan  for  the  required 
services. 

4.  Combined  DT&E  and  OT&E 

The  USN  has  a  policy  that  combined  DT&E  and  OT&E  testing  should  only 
be  considered  when  there  are  time  and  cost  savings  [Ref.  24:p.  8-3].  However,  this 
combined  approach  must  not  compromise  either  the  developmental  or  operational 
test  objectives.  Also  a  final  independent  phase  of  operational  testing  and  evaluation, 
termed  “OPEVAL”,  is  required  for  beyond  low  rate  initial  production  decisions 
[Ref.  24:  p.  8-4]. 

“Combined  testing”  refers  to  a  single  test  program  conducted  to  support 
both  DT&E  and  OT&E  objectives.  The  advantages  of  combined  testing  are  the 
shorter  time  required  for  testing,  and  cost  savings  by  eliminating  redundant 
activities.  These  need  to  be  weighed  against  the  limitations  of  the  additional 
extensive  coordination  required  and  the  less  than  optimum  environment  and 
coverage  for  OT&E  that  may  occur.  Early  involvement  of  OT&E  personnel  during 
system  development  increases  their  familiarity  with  the  system  and  permits 
identification  of  operational  concerns  early  in  the  program. 
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The  U.S.  has  extensive  experience  with  combined  testing,  some  successful 
and  some  unsuccessful,  and  they  conclude  that  it  is  possible  to  have  combined  test 
teams  involved  throughout  the  testing  process.  The  DT&E  and  OT«feE  teams  can 
share  mutually  beneficial  data,  as  long  as  the  test  program  is  carefully  planned, 
evaluated  and  reporting  activities  are  conducted  separately.  [Ref.  18:pp.  17-1  -  17- 
4]. 

5.  Modeling  and  Simulation 

DoD  directives  encourage  the  use  of  simulation  and  modeling  to  assist  in 
projecting  operational  effectiveness  and  operational  suitability  prior  to  Milestone  II, 
but  limit  their  use  in  subsequent  OT&E  to  that  of  supplementing  OT&E  test  data. 
The  use  of  modeling  and  simulation  can  increase  the  efficiency  of  the  T&E  process, 
reduce  the  time  and  cost,  provide  otherwise  unattainable  and  unmeasureable  data, 
and  provide  more  timely  and  valid  results.  Although  simulations  are  not  a  substitute 
for  live  testing,  USN  OT&E  policy  recognises  that  it  is  useful  to  augment  tests  by 
simulating  non-testable  events  and  scenarios,  and  for  overcoming  resource 
limitations.  Simulation  can  also  be  used  to  extend  test  results,  to  improve  the 
statistical  sample,  or  to  determine  overlooked  or  directly  unmeasured  parameters. 
[Ref.  18:pp.  16-1  -  16-8] 

6.  Foreign  Weapons  Evaluation 

The  US  DoD  has  a  foreign  weapons  evaluation  program  which  is  designed 
to  support  the  evaluation  of  a  foreign  nation’s  weapons  system,  equipment  or 
technology  in  terms  of  its  potential  to  meet  a  specific  U.S.  military  requirement.  The 
primary  objective  of  the  program  is  to  reduce  the  costs  of  U.S.  research  and 
development,  while  leading  to  the  acquisition  of  foreign  equipment  for  U.S.  use 
[Ref.  18.p.  21-1].  From  the  OT&E  viewpoint,  the  USN  ‘try-before-buy  policy’  is 
still  maintained,  despite  the  early  phases  of  OT  being  unable  to  be  achieved. 
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COMOPTEVFOR  may  be  directed  to  assess  the  adequacy  of  any  previously 
conducted  OT&E  and  to  provide  recommendations  on  the  need  for  additional  U.S. 
T&E  prior  to  procurement  [Ref.  27:p.  6].  Obtaining  pertinent  test  and  evaluation 
data  from  foreign  governments  and  manufacturers  is  necessary  to  preclude 
duplication  and  reduce  costs. 

7.  OT&E  Funding 

Congress  mandates  that  the  costs  of  initial  OT&E  required  to  achieve  a  full 
production  decision  shall  be  paid  from  funds  available  for  the  system  being  tested 
[Ref.  17].  Funding  associated  with  T&E  (including  instrumentation,  targets  and 
simulations)  are  identified  in  the  system  acquisition  cost  estimates,  acquisition  plans 
and  the  TEMP  [Ref.  18:p.  18-11].  The  Program  Manager  plans,  programs,  budgets 
and  funds  the  costs  of  ail  resources  identified  in  the  approved  TEMP  for  all  T&E 
through  OT-III.  OPTEVFOR  estimates  the  costs  to  conduct  OT&E  and  the  program 
manager  budgets  and  funds  these  costs.  OT&E  costs  include  test  articles, 
expendables,  targets,  data  collection  and  reduction  and  OPTEVFOR  program 
related  costs.  The  Program  Manager  does  not  fund  fleet  operating  costs  for  T&E 
support,  which  includes  fuel  and  aircraft.  The  operating  costs  for  OT-II  and  III,  and 
all  costs  for  OT-IV,  except  procurement  costs  and  OPTEVFOR  costs,  are  funded  by 
the  Fleet  CINCs.[Ref.  ll:p.  21]  OPTEVFOR  internal  man-hours,  and  computer 
hours  actually  expended  are  not  charged  to  individual  programs. 

8.  Land  Based  Test  Sites 

A  Land  Based  Test  Site  (LBTS)  is  a  facility  that  duplicates,  simulates  or 
stimulates  the  employment  of  a  system’s  planned  operational  installation  and 
utilization,  primarily  for  the  purpose  of  conducting  DT&E.  LBTS  are  often  used  to 
test  system  integration  and  overall  performance.  [Ref.  1  l:p.  12]  COMOPTEVFOR 
advises  the  CNO  on  the  adequacy  of  LBTS  for  the  conduct  erf  OT&E  LBTS  test  data 
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is  normally  used  to  support  Milestone  IIA  decisions  and  not  the  more  stringent 
Milestone  III.  Except  where  approved  by  CNO,  OT&E  intended  to  support 
production  decisions  will  be  performed  in  the  operational  environment  in  preference 
to  the  LETS. 

9.  Ship  OT&E 

a.  But  Ships  are  Different 

The  accomplishment  of  ship  T&E  varies  considerably  from  the  normal 
test  cycle  due  to  the  lengthy  period  for  design,  engineering  and  construction  of  a 
major  ship,  and  because  ship  T&E  includes  both  that  conducted  on  the  ship  platform 
itself,  as  well  as  that  conducted  on  the  equipment  and  systems  to  be  installed  on  the 
ship. 

Ship  acquisitions  are  low  volume,  high  cost  programs,  and  so,  while 
subject  to  the  same  basic  DoD  and  USN  T&E  policies  applied  to  other  systems,  their 
procurement  requires  special  T&E  processes.  If  ships  were  procured  the  way  other 
Navy  systems  are,  the  lead  ship  of  the  class  would  be  used  as  a  prototype  for 
conducting  T&E  prior  to  approving  the  construction  of  the  follow-on  ships.  Because 
of  the  time  associated  with  the  design  and  construction  of  a  ship  and  the  fact  that  few 
technical  or  operational  risks  are  associated  with  the  ship  platform  itself,  the 
prototyping  approach  in  not  necessary. 

In  a  typical  ship  acquisition  program,  it  can  take  about  five  years 
between  the  contract  award  for  the  lead  ship  and  the  time  the  ship  is  itself  ready  to 
conduct  at-sea  operational  testing.  To  delay  the  construction  of  the  follow-on  ships 
would  have  a  significant  impact  on  the  shipbuilding  program’s  costs  and  schedule. 
This  longer  schedule  would  substantially  increase  shipyard  costs  in  return  for  a 
negligible  reduction  in  the  risk  of  the  ship  not  meeting  operational  requirements.  A 
significant  time  lag  between  the  production  of  the  lead  and  follow-on  ships  would 
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force  a  shipbuilder  to  let  most  of  his  experienced  workers  go  and  rehire  and  retrain 
others  when  construction  resumed.  Moreover,  the  significant  cost  savings  available 
through  quantity  procurement  of  many  of  the  ships’  equipments  would  not  be 
realised  if  there  was  a  time  lag  in  production  between  the  lead  and  follow-on 
ships.  [Ref.  29]  Also,  from  a  contracting  standpoint,  a  significant  amount  of  lead 
time  is  required  so  that  equipment  will  be  available  when  needed  for  construction. 
Much  of  the  long  lead  equipment  is  contracted  before  there  is  a  ship  available  to 
test.  [Ref.  19] 

b.  Policies  and  Principles 

Congress  accepted  these  practicalities  of  ship  OT&E  and  amended  the 
law  to  recognise  the  special  case  of  ship  OT&E  [Ref.  30].  Because  the  development 
and  construction  period  for  a  major  ship  normally  precludes  completion  of  initial 
OT&E  on  the  lead  ship  prior  to  the  production  decision  for  follow-on  ships, 
successive  phases  of  lOT&E  are  accomplished  as  soon  as  practicable  to  reduce  risk 
and  minimise  the  need  for  modification  to  follow-on  units. 

(1)  The  OT-I  phase  is  an  Eariy  Operational  Assessment  (EOA)  and  is 
an  evaluation  of  technologies,  of  processes,  of  plans  and  procedures,  and  design 
details  to  see  if  there  any  high  risk  areas  that  have  a  potential  to  cause  cost  overruns 
or  degrade  performance. 

(2)  The  OT-II  phase  for  a  shipbuilding  program  is  an  extension  of  this 
concept,  except  that  it  occurs  in  a  time  frame  where  landbased  testing  facilities  or 
mock-ups  of  sections  exist.  Operability  and  suitability  can  be  assessed,  though  not 
necessarily  on  a  production  representative  item. 

Ship  acquisition  programs,  therefore,  usually  have  lOT&E  between 
Milestones  II  and  III,  which  consist  of  individual  weapon  systems  testing  and 
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system  integration  at  land  based  test  sites.  A  total  ship  OPEVAL  is  impractical  in  all 
cases  except  programs  where  a  lead  ship  is  constructed  as  an  R&D  effort  involving 
a  major  technological  advance  in  hull  or  machinery  design  [Ref.  28]. 


F.  SHIP  ACCEPTANCE 
1.  Policy 

The  purpose  of  ship  acceptance  in  the  USN  is  to  ensure  delivery  to  the  Fleet 
of  complete  ships,  free  from  both  contractor  and  government  responsible 
deficiencies  [Ref.  35:p.  1].  Ship  acceptance  has  traditionally  been  based  only  on  the 
successful  completion  of  PAT&E  and  material  inspections,  and  for  the  first  of  class, 
has  culminated  in  approval  for  Fleet  Introduction.  However,  with  the  treatment  of 
ships  as  complete  systems,  as  for  the  DDG-Sl,  Fleet  Introduction  was  based  also  on 
OT&E  results. 

Z  Definitions 

Reference  35  provides  the  major  definitions  for  the  acceptance  process: 

•  “Acceptance”  is  defined  as  the  legal  act  of  accepting  custody  of  a  new 
construction  ship  by  the  Navy  upon  delivery  of  the  ship  by  a  private  builder. 

•  “Delivery”  is  defined  as  the  actual  assumption  of  custody  by  the  Navy 
incident  to  acceptance.  The  date  of  delivery  from  a  private  shipyard  is  also 
the  date  of  acceptance. 

•  “Fleet  Introduction”  is  more  a  concept  than  a  specifically  defined  milestone  K 
It  signifies  approval  by  the  Secretary  of  the  Navy  that  a  ship  class  meets  the 
operational  effectiveness,  operational  suitability,  safety  and  material 
standards  for  service  use. 


1 .  Coovcnaiioo  between  C^itain  Tobin.  USN.  Surface  Division.  Board  of  Inspecdon  and  Survey, 
and  tbe  autbor.  27  October.  1992. 
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3.  Responsibilities 

Independent  verification  of  readiness  of  ships  for  acceptance  is  the 
responsibility  of  the  President,  Board  of  Inspection  and  Survey  (PREINSURV)  who 
conducts  Acceptance  Trials  and  Final  Contract  Trials  for  each  individual  new  ship 
[Ref.  1  l:p.  10].  Based  upon  its  findings,  the  Board  recommends  acceptance  or  final 
settlement  of  the  contract.  The  Board  identifies  material  conditions  which  represent 
departures  from  the  General  Specifications  and  deficiencies  that  substantially  reduce 
the  ship’s  fitness  for  naval  service  and/or  degrade  its  ability  to  perform  its  primary 
mission.  [Ref.  34:p.  1]  These  deficiencies  are  noted  in  the  Ship’s  Log,  but  may  not 
necessarily  be  corrected. 

A  ship  is  normally  accepted  by  the  Commander,  Naval  Sea  Systems 
Command.  The  preparation  and  presentation  of  a  ship  for  acceptance  trials  is  the 
responsibility  of  the  Supervising  Authority,  normally  the  Supervisor  of  Shipbuilding 
(SUPSHIPS).  Users  also  have  a  say  in  the  acceptance  process.  A  ship’s  prospective 
Commanding  Officer  submits  progress  reports  and  can  request  changes  be  made 
which  are  essential  to  safety  or  the  ship’s  mission.  The  Type  Commander  monitors 
the  construction  and  acceptance  process  to  ensure  “customer”  input  is  provided, 
conducts  a  pre-commissioning  habitability  inspection  and  makes  an  acceptance 
recommendation  to  the  Fleet  Commander  who,  in  turn,  makes  his  recommendation 
on  the  request  to  deliver  the  ship<[Ref.  35:pp.  2  -  6] 

4.  Procedure 

The  procedure  for  ship  acceptance  and  introduction  to  the  Fleet  is  outlined 
in  Figure  5.  The  Total  Ship  Test  Program  outlines  two  phases  of  this  process  [Ref. 
32:p.  2-6]: 

•  Ship  Construction  Tests  and  Trials  consisting  of  all  testing  conducted  during 
construction,  including  Acceptance  Trials. 


Ship  Post-Delivery  Test  and  Trials  include  final  contract  trials  that 
commence  after  ship  delivery  and  continue  to  the  end  of  the  ship  construction 
obligation. 


Figure  5:  Traditional  Ship  Acceptance  Schedule  and  Milestones 


This  procedure  includes  the  following  activities  and  milestones: 
a.  Delivery  I  Acceptance 

Acceptance  Trials  are  conducted  by  the  INSURV  Board  when  all  work 
has  been  completed.  These  trials  are  conducted  at  sea  and  in  port  when  the  following 
prerequisites  are  met  [Ref.  35;p.  11]; 

•  Successful  completion  of  builders  trials. 

•  All  installed  equipment  operable  and  capable  of  meeting  performance 
specifications. 

•  Habitability  items  complete. 

•  Completion  of  surveys  not  requiring  remote  ranges. 

•  Installation  and  checkout  tests  completed. 
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•  All  applicable  naval  certifications  completed. 

•  Completed  test  reports  and  certifications  available  for  inspection  by  the  Trial 
Board. 

On  successful  trial  completion,  and  on  the  recommendation  of 
PREINSURV,  COMNAVSEA  accepts  the  ship  on  behalf  of  the  USN  from  the 
contractor. 

b.  Final  Contract  Trials 

Final  Contract  Trials  are  trials  and  materiel  inspections  conducted  at 
sea  and  in  port  by  the  INSURV  Board  to  determine  if  builder  responsible 
equipments  are  operating  satisfactorily  [Ref.  35;p.  E3].  It  involves  operation  of  ail 
combat  systems,  and  the  propulsion  system  at  full  power.  On  completion, 
PREINSURV  submits  a  technical  assessment  of  readiness  for  OT&E  to  CNO,  and 
to  COMOPTEVFOR  when  tasked  by  CNO  [Ref.  1 1  :p.  10]. 

c.  Fleet  Introduction 

The  successful  completion  of  Final  Contract  Trials  on  the  first  ship  of 
the  class  traditionally  results  in  a  recommendation  for  Fleet  Introduction  by 
PREINSURV  to  CNO.  The  Secretary  of  the  Navy  then  approves  Fleet  Introduction 
of  the  ship  class  on  the  recommendation  of  CNO.  In  contrast.  Fleet  Introduction  of 
systems  is  traditionally  recommended  by  OPTEVFOR  after  successful  completion 
of  OT&E.  However,  for  the  DDG-51  class,  where  the  ship  was  viewed  as  a  complete 
system.  Fleet  Introduction  was  recommended  by  both  PREINSURV  and 

OPTEVFOR^after  the  completion  of  OT&E.  Figure  6  show  the  acceptance  and 
schedule  milestones  used  for  DDG'51. 


2.  Conversatioa  with  Conunaiicler  J.  Kien,  USN,  OPTEV’FOR,  and  the  author  29  OcttAer,  1992. 
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Figure  6:  DDG-51  Ship  Acceptance  Schedule  and  Milestones 


5.  Ship  Acceptance  and  OT&E 

OT&E  has  traditionally  not  had  an  input  into  the  ship  acceptance  process  in 
the  USN,  with  no  involvement  in  Delivery,  Acceptance  or  Fleet  Introduction.  It 
primarily  focussed  on  the  ship’s  systems  rather  than  the  ship  as  a  whole,  and  was 
fundamental  to  the  Fleet  Introduction  of  these  systems.  With  the  view  of  the  DDG- 
51  as  a  complete  system,  successful  completion  of  OT&E  is  now  a  prerequisite  for 
the  Fleet  Introduction  of  all  new  warship  classes. 

G.  USN  OT&E  PROCEDURES 

Procedures  are  the  methodology  by  which  policy  is  carried  out.  This  section 
discusses  the  procedures  adopted  by  the  USN  in  implementing  OT&E  policy. 

1.  From  the  User  Requirement  to  the  OT&E  Test  Plan 

The  determination  of  the  level  and  extent  of  OT&E  required  is  a  difficult 
question.  For  their  multiwarfare  capable  ships,  the  USN  focuses  on  the  mission 
warfare  areas  to  determine  what  is  appropriate  [Ref.  19].  The  fundamental 
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cornerstone  of  OT&E  is  the  user  requirement,  generated  through  a  series  of 
documents  built  into  the  acquisition  process.  The  procedure  involves  progression 
through  the  following  documents  (Figure  7): 

•  Mission  Need  Statement  (MNS) 

•  Cost  And  Operational  Effectiveness  Analysis  (COEA) 

•  Operational  Requirements  Document  (ORD) 

•  Top  Level  Requirement  (TLR) 

•  Test  and  Evaluation  Master  Plan  (TEMP) 

•  OT&E  Test  Plan  (OT&E  TP) 


Figure  7:  Progressive  Definition  of  OT&E  Documentation 
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a.  Mission  Need  Statement 

Acquisition  programs  are  based  on  identified  mission  needs  that  are 
generated  as  a  direct  result  of  continuing  assessments  of  current  and  projected 
capabilities  in  the  context  of  changing  military  threats  and  Defence  policy.  A  broad 
statement  of  mission  need,  expressed  in  terms  of  an  operational  capability,  not  a 
system-specific  solution,  is  identified  in  a  Mission  Need  Statement  (MNS).  [Ref. 
24:p.  3-2] 

b.  Cost  and  Operational  Effectiveness  Analysis 

A  cost  and  operational  effectiveness  analysis  (COEA)  may  be 
undertaken  to  evaluate  the  costs  and  benefits  of  alternative  courses  of  action  to 
satisfy  the  requirements  of  the  MNS.  The  COEA  looks  at  other  alternatives  e.g.,  are 
a  lot  of  little  ships  or  a  few  big  ships  needed  to  do  the  mission?  The  most  suitable 
alternative  is  selected,  e.g.,  a  lot  of  little  ships  are  required,  and  they  need  to  do  these 
things.  It  also  develops  the  Measures  of  Effectiveness  (MOE)  for  assessing  the 
operational  effectiveness  of  the  alternatives. 

c.  Operational  Requirements  Document 

From  the  MNS  and  COEA  is  developed  the  Operational  Requirements 
Document  (ORD)  that  contains  the  performance  (operational  effectiveness  and 
operational  suitability)  and  related  operational  parameters  for  the  proposed  concept 
or  system.  It  contains  the  system  performance  objectives  and  minimum  acceptable 
requirements  to  meet  the  requirements  of  the  MNS.  The  MOEs  used  in  the  ORD  and 
the  technical  parameters,  are  designed  to  support  those  same  MOEs  that  were 
developed  in  the  COEA  [Ref  19]. 
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d.  Top  Level  Requirement 

For  some  ship  acquisition  programs.  Top  Level  Requirements  are 
prepared  after  the  MNS  and  ORD.  This  is  necessary  because  of  the  length  and 
complexity  of  the  ship  design  process.  The  TLR  states  the  high  level  requirements 
for  the  ship.  For  example,  the  TLR  for  the  DDG-51.  Section  2.1  la  Operational 
Directions,  under  Mission,  Warfare  Areas  states: 

Simultaneous  action  in  Strike  Warfare,  Antisubmarine  Warfare,  Antisurface 
Warfare  and  Antiair  Warfare  required.  The  combat  system  shall  provide  for 
rapid,  accurate  and  efficient  employment  of  the  ship’s  weapons. 

e.  Test  and  Evaluation  Master  Plan 

The  TEMP  is  developed  from  the  ORD,  and  TLR  if  produced,  and  is 
the  controlling  T&E  document  that  defines  and  integrates  the  test  objectives,  and 
Critical  Operational  Issues  (COI)  based  on  the  ORD.  For  example,  the  DDG-51 
TEMP  [Ref.  31:p.  IV-2]  highlights  the  COI  of  Combat  System  Performance  as 
follows: 

Will  the  combat  systems  support  simultaneous  action  against  threats  in  Antiair 
Warfare  (AAW),  Antisubmarine  Warfare  (ASW),  Antisurface  Warfare 
(ASUW),  and  Strike  Warfare  (STW)  missions  during  independent  or 
combined  operations  as  a  unit  of  a  Battlegroup,  Surface  Action  Group, 
Underway  Replenishment  Group  or  Amphibious  Task  Force? 

This  COI  is  addressed  as  a  specific  objective  for  OT-IIIA  [Ref.  3I:p. 
IV-11]  which  is  to: 

Determine  the  capability  of  the  integrated  DDG-51  combat  system  to  detect, 
track,  localise  and  engage  threat  representative  targets  in  a  multiwarfare 
environment  (STW,  ASW,  ASU  and  AAW). 


From  this  specific  objective,  a  test  procedure  is  outlined  by 
OPTEVFOR.  The  level  and  extent  of  testing  is  based  on  the  level  of  confidence  to 
be  assessed.  It  is  usually  a  compromise  between  the  extent  of  testing  and  funding 
available,  agreed  between  the  PM  and  OPTEVFOR  with  DOT&E  oversight.  The 
TEMP  is  viewed  as  a  contract  between  the  Developer,  the  Sponsor,  OPTEVFOR 
and  OSD,  through  DOT&E,  as  what  is  committed  to  be  done  and  funding  is 
available.  Any  change  to  the  TEMP  must  be  approved  by  all  signatories.  The  TEMP 
identifies  limitations  to  the  scope  of  testing  which  may  include  threat  replicators 

e.g.,  when  a  MIG-29  threat  aircraft  is  needed  during  testing,  but  an  F-14  will  be 
used.  [Ref.  19] 

f.  OT&E  Test  Plan 

This  specific  objective  is  then  followed  through  to  the  OT&E  Test  Plan, 
developed  by  COMOPTEVFOR,  The  test  plan  contains  “E”  tests  that  assess 
operational  effectiveness  and  “S”  tests  that  assess  operational  suitability.The  test 
plan  then  addresses  how  this  will  be  done,  by  breaking  it  down  into  more  specifics, 
AAW,  EW,  CAP,  CIWS.  Under  DDG  Test  E-1  Combat  System  Performance,  the 
Test  Plan  states: 

To  determine  the  capability  of  the  integrated  DDG-51  combat  system  to 
detect,  track,  localize  and  engage  threat  representative  targets  in  a 
multiwarfare  environment  (STW,  ASW,  ASU  and  AAW). 

g.  Summary 

A  requirement  can,  therefore,  be  traced  from  the  mission  need  to  the 
ORD,  through  to  a  COI  and  specific  a  objective  in  the  TEMP,  to  the  test  plan  and 
ultimately  to  the  final  OT&E  report  by  OPTEVFOR.  This  linkage  from  mission 
need  to  operational  test  program  defines  the  scope  of  testing.  If  a  ship  has  a  number 
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of  mission  areas  and  there  are  MOEs  to  evaluate  the  performance  in  those  mission 
areas,  then  a  test  program  is  developed  around  those  MOEs. 

2.  Assessment  of  Operational  Effectiveness 

In  assessing  operational  effectiveness,  OPTEVFOR  examines  each  COI 
and  related  operational  effectiveness  objective,  and  decides  what  needs  to  be  known 
to  enable  each  objective  to  be  assessed  [Ref.  15:p.  9-1].  For  example,  the  objective 
may  be  “to  determine  the  sonar’s  capability  to  detect,  classify,  and  track...  in  the 
natural  acoustic  environment.”  OPTEVFOR  may  decide  the  following  measures  of 
effectiveness  are  needed  to  assess  the  overall  objective: 

•  Probability  of  detection 

•  Detection  range 

•  Probability  of  correct  classification  given  detection 

•  Probability  of  classifying  a  threat  as  a  nonthreat 

•  Time  between  detection  and  classification 

•  Classification  Range 

•  Probability  of  establishing  a  track,  given  detection 

•  Time  between  detection  and  track  establishment 

•  Range  at  track  establishment 

•  Percent  of  time  tracks  are  held 

They  ensure  that  the  appropriate  environments,  threats,  etc.,  are  included 
and  that  sufficient  data  will  be  generated  to  address  the  COI  and  objectives. 
OPTEVFOR  initially  determines  the  requirements  and  resources  necessary  to 
conduct  the  test.  This  is  achieved  by  making  a  list  of  environments  and  other 
conditions  under  which  the  system  will  operate,  and  associating  them  with  the 
various  threats.  This  can  be  achieved  by  constructing  a  test  design  matrix  of  which 
Table  5  is  an  example. 
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TABLE  5:  SAMPLE  TEST  DESIGN  MATRIX  [Ref.  15:p.  H-2] 


THREAT 

GROUP 

CHAR 

TEST 

TARGET 

BENIGN 

ECM 

ECM 

(SOJ/SSJ) 

CHAFF 

(SRBOC) 

SEASTATE 

(1-2) 

MULTI 

TARGET 

TOTAL 

MISSILE 

I 

n 

SEASKIM 

(<10in) 

SUBSONIC 

MOD 

BQ.M-34S 

A 

B 

C 

C 

ABC 

B 

3 

I 

IV 

10-60in 

SUBSONIC 

BQM-34S 

J 

E 

El 

2 

in 

<30m 

SUPERSONIC 

MOD 

MQM-8G 

D 

D 

D 

1 

in 

VI 

30-60m 

SUPERSONIC 

MQM-8G 

H 

F 

FH 

in 

VI 

60-300m 

SUPERSONIC 

MQM-8G 

0 

in 

25-50DEG 

DIVE 

SUPERSONIC 

MQM-8G 

0 

I 

10.60in 

SUBSONIC 

QF-86 

G 

G 

1 

n 

SURFACE 

QST-35 

I 

1 

The  letters  indicate  missile  firing  runs  in  alphabetical  order  of  priority.  The 
columns  are  the  environment  in  which  the  firing  is  scheduled.  [Ref.  15:p.  H-1]  Then 
the  actual  resources  available  are  reviewed  to  determine  which  test  objectives  must 
be  eliminated  due  to  lack  of  funding. 

Even  though  the  operational  test  program  is  linked  to  mission  need  through 
the  COEA,  ORD  and  TEMP,  there  are  a  number  of  difficulties  in  developing 
appropriate  OT&E  tests.  Many  operational  requirements  are  not  very  specific, 
meaning  the  testing  and  subsequent  assessment  of  those  requirements  may  be  very 
subjective,  and  critical  definitions  need  to  be  agreed  before  testing  commences. 
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a.  Qualitative  Vs.  Quantitative  Assessment 

The  broad  scope  of  many  operational  requirements  makes  their  testing 
and  subsequent  assessment  rather  subjective,  making  the  determination  of 
meaningful  and  assessable  quantitative  MOEs  difficult.  Also  a  lack  of  resources 
may  reduce  an  objective,  quantitative,  and  statistically  relevant  test  schedule  to  a 
more  subjective,  qualitative  assessment.  These  situations  require  expertise  and 
judgement,  and  compromise  between  the  authorities  involved,  to  make  the  tests  as 
objective  as  possible.  As  an  example,  an  antisurface  warfare  mission  may  require 
the  ability  to  sink  another  ship,  but  the  test  program  does  not  include  this  provision. 

The  test  program  has  to  be  designed  so  that  it  will  provide  confidence 
that  the  ability  can  be  demonstrated  without  actually  sinking  a  ship.  Similar 
situations  are  encountered  in  all  the  warfare  areas,  and  additionally  with  ship 
survivability.  One  objective  is  to  achieve  statistical  relevance.  DOT&E  works 
closely  with  the  USN  to  help  determine  how  many  test  assets  are  required  to  achieve 
a  certain  degree  of  confidence  that  a  test  objective  has  been  met.  A  test  may  need  to 
be  repeated  a  number  of  times  to  gain  statistical  relevance.  However,  if  this  is  not 
possible,  or  the  direct  proving  of  an  ability  is  not  possible  (as  in  the  surface  warfare 
example),  then  the  test  needs  to  be  designed  so  that  what  is  observed  in  the  test 
program  is  projected  to  be  what  would  be  observed  in  reality. 

OPTEVFOR  indicates  the  statistical  significance  of  a  result  as  follows: 

•  ‘‘Determine”  means  a  statistically  significantly  result  with  a  specified  level  of 
confidence. 

•  ‘‘Assess”  means  to  qualitatively  evaluate  which  will  not  achieve  a 
statistically  significantly  result. 
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b.  Definition  Agreement 

Another  difficulty  in  assessing  operational  effectiveness  is  in  defining 
the  meaning  of  the  requirement  and  the  measures  of  effectiveness.  This  is 
particularly  applicable  to  whole  ship  testing  which  may  incorporate  a  specific 
equipment  requirement  in  addition  to  an  overall  ship  requirement.  For  example,  the 
definition  of  a  “Detection  Opportunity”  for  a  sonar,  may  differ  from  what  is  an  ASW 
“Detection  Opportunity”  for  the  ship,  given  the  other  ASW  related  sensors  that  may 
be  available.  Agreement  between  authorities  and  the  MNS  is  required  for  the  whole 
ship  versus  individual  systems.  [Ref.  21] 

3.  Assessment  of  Operational  Suitability 

The  assessment  of  operational  suitability  in  the  USN  is  more  standardised 
than  for  the  assessment  of  operational  effectiveness.  OPTEVFOR  identities  14 
suitability  issues  for  which  they  have  standard  suitability  tests  [Ref.  15:p.  9-4  -  9-8]: 

•  Reliability 

•  Maintainability 

•  Availability 

•  Logistic  supportability 

•  Compatibility 

•  Interoperability 

•  Training 

•  Human  factors 

•  Safety 

•  Documentation 

•  Transportability 

•  Wartime  usage  rates 

•  Manning 

•  Software  supportability 


Software  supportability  has  only  recently  been  added  by  OPTEVFOR.  It 
relates  particularly  to  totally  intensive  computer  systems,  termed  “software 
intensive  systems”.  It  determines  if  a  program  manager  has  established  adequate 
software  support  for  his  system. [Ref.  21]  Not  all  these  suitability  issues  need  be 
addressed  for  a  particular  system.  The  nature  and  use  of  the  system,  and  the  phase 
of  OT&E  will  determine  the  issues  to  be  assessed. 

€L  Suitability  Assessment  Procedure 

Early  in  the  acquisition  program,  OPTEVFOR  reviews  the  Integrated 
Logistic  Support  Plan  (ILSP)  against  the  ORD.  The  ILSP  is  a  key  acquisition 
document  which  defines  the  various  methods  used  to  provide  the  required  range  of 
logistic  support  for  the  system.Following  ILSP  review,  OPTEVFOR  determines  the 
suitability  tests  required  based  on  expected  reliability,  degree  of  confidence, 
thresholds,  etc.  required  to  be  achieved.  These  determine  the  scope  and  length  of  the 
assessment.  An  onboard  assessment  at  sea  is  conducted,  during  which 
questionnaires  of  crew  members  may  be  used  and  maintainability  demonstrations 
are  performed,  usually  using  pre-faulted  modules.  Reliability,  maintainability  and 
availability  (RM&A)  data  is  collected  and  sent  to  the  normal  inservice  analyst  of 
RM&A  data,  the  Naval  Weapons  Analysis  Centre  (NWAC)  for  analysis,  in  later 
OT&E,  OPTEVFOR  also  uses  data  from  the  inservice  maintenance  management 
system  (3M)  and  the  individual  system  logs.  [Ref.  21] 

Not  only  does  OPTEVFOR  have  specialist  analysts  for  effectiveness 
issues,  it  also  has  suitability  analysts  who  design  tests  and  determine  measures  of 
suitability.  They  also  evaluate  and  analyse  the  adequacy  of  logistic  supportability. 
OPTEVFOR  also  relies  on  operational  personnel  to  use  their  experience  and 
knowledge  of  the  system  to  identify  inadequate  logistic  support. 
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4.  OT&E  Reporting 


a.  Reporting  by  COMOPTEVFOR 

COMOPTEVFOR’s  evaluation  report  provides  the  CNO  with 
conclusions  regarding  a  system’s  operational  effectiveness  and  operational 
suitability,  and  his  recommendations  regarding  the  systems  future,  i.e.,  fleet 
introduction,  further  development,  additional  OT&E,  etc.  The  report  also  contains 
the  test  results  and  evaluation  criteria  to  substantiate  the  conclusions  and 
recommendations.  [Ref.  15:p.  12-1]  A  report  covers  a  complete  OT&E  phase  (e.g., 
OT-IIA),  relating  the  test  results  to  the  COIs  and  addressing  the  objectives  stated  in 
the  TEMP.  COMOPTEVFOR  requests  comments  from  the  Commanding  Officer  of 
the  ship  under  test  ship.  These  comments  are  sent  only  to  OPTEVFOR.  All 
operational  test  data  is  considered  to  be  the  owned  by  OPTEVFOR  until  the  final 
report  is  signed.  Quick-look  and  interim  reports  are  usually  sent  only  if  the  testing 
could  not  be  ccxnpleted  or  when  directed  by  the  CNO.  Although  the  final  report  may 
be  a  surprise  to  other  authorities,  the  USN  appears  to  favor  this  reporting  procedure 
to  ensure  that: 

•  the  conduct  of  OT&E  remains  impartial  and  is  not  influenced  by  the  program 
manager  or  contractors  during  testing  and  analysis. 

•  OPTEVFOR’s  conclusions  are  based  on  a  complete  and  thorough  analysis. 

•  OT&E  results  and  analysis  are  reported  through  one  authority  only. 

b.  Reporting  by  DOT &E 

The  reports  rendered  by  DOT&E  address  whether  the  OT&E 
performed  was  adequate  and  whether  the  OT&E  results  confirm  that  the  system 
actually  tested  is  operationally  effective  and  operationally  suitable.  In  the  past  it 
used  OPTEVFOR  reports  as  basis  and  provided  their  own  Judgement.  Following 
General  Accounting  Office  (GAO)  criticism  that  DOT&E  overall  assessments 
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consistently  presented  a  more  favorable  presentation  to  the  Congress  of  test 
adequacy  and  system  performance  than  was  warranted  by  the  facts  [Ref.  36:p.  31], 
it  now  does  an  independent  analysis  from  the  raw  test  data  collected  by 
OPTEVFOR.  As  DOT&E  does  not  have  any  specialist  analysis  staff,  it  hires  an 
independent  contractor  who  works  exclusively  for  DOT&E,  to  do  the  analysis. 
Experienced  operators  at  DOT&E  then  make  the  assessment.  DOT&E’s  written 
report  to  Congress  is  based  now  on  OPTEVFOR  reports  and  its  own  independent 
analysis. [Ref.  21] 

5.  OT&E  Coordination 

The  USN  recognises  the  importance  of  coordination  between  authorities  in 
the  successful  achievement  of  T&E.  The  USN  T&E  coordinator  is  responsible  to  the 
Director,  Test,  Evaluation  and  Technology  Requirements  (N091).  He  is  essentially 
the  OPNAV  area  coordinator  for  Navy  T&E,  providing  a  primary  contact  point  for 
all  parties  and  setting  up  T&E  briefings  and  meetings.  His  responsibilities  include 
the  chairmanship  of  the  Test  and  Evaluation  Coordination  Group  (TECG)  for  each 
major  program.  [Ref.  37]  TECGs  are  used  for  complex,  multifaceted  programs 
which  require  extensive  T&E  coordination.  Membership  of  a  TECG  includes  the 
Program  Manager,  the  Sponsor,  COMOPTEVFOR,  a  logistics  coordinator  and 
others  as  appropriate  (such  as  a  PREINSURV  representative).  TECG 
recommendations  are  considered  for  inclusion  in  the  TEMP. [Ref.  1  l:p.  15]  Some  of 
the  functions  of  a  TECG  are  the  early  definition  of  terms,  measures  of  effectiveness 
and  how  these  are  to  be  measured,  and  the  criteria  for  acceptable  or  not  acceptable. 
Of  note  is  the  formation  of  a  TECG  does  not  imply  a  Joint  test  team  approach.  Each 
T&E  agency  remains  fully  and  solely  responsible  for  conducting  and  reporting  the 
types  and  phases  of  T&E  for  which  it  is  accountable. 


<S.  OT&E  Personnel  Selection  and  Training 

OT&E  is  a  specialised  discipline,  with  its  own  philosophy  and 
methodology.  The  U.S.  community  involved  with  ship  OT&E  select  and  train  their 
military  personnel.  Although  some  personnel  may  develop  into  OT&E  specialists, 
no  career  path  in  OT&E  is  consciously  provided,  and  no  naval  personnel  currently 
serving  in  DOT&E  have  served  previously  in  OPTEVFOR. 

a.  OPTEVFOR 

There  are  no  special  selection  requirements  for  detailing  military 
personnel  for  service  as  Test  Directors  in  OPTEVFOR,  although  broad  ship 
operations  experience  and  combat  system  knowledge  are  preferred.  OPTEVFOR  is 
functionally  organised  along  warfare  lines,  so  specialists  in  each  warfare  area  are 
preferred.  Most  Test  Directors  are  of  Commander  rank,  reporting  to  Captain  level 
section  heads.  For  particular  programs,  OPTEVFOR  makes  use  of  subject  area 
experts,  either  resident  or  borrowed  from  a  non-interested  party.  OPTEVFOR 
conducts  an  intensive  four  day  Operational  Test  Directors  (OTD)  course  covering 
the  major  areas  of  OT&E.  Further  details  of  this  course  are  included  in  Appendix  A. 
They  also  run  adhoc  segment  courses  which  provide  acquaints  or  updates  on  OT&E 
subjects  e.g.,  analysis,  test  plan  development  and  threat  updates.  To  assist  their 
personnel  in  managing  OT&E,  OPTEVFOR  publishes  the  OT&E  Director’s  Guide 
which  documents  its  philosophy  and  methodology.  [Ref.  21] 

b.  DOT&E 

Military  personnel  assigned  to  DOT&E  are  selected  by  background  and 
expertise,  and  are  preferably  war  college  graduates  with  joint  experience.  They  are 
interviewed  by  their  potential  immediate  superior,  and  by  DOT&E  himself,  before 
their  appointment  is  confirmed.  DOT&E  is  platform,  rather  than  functionally 
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organised,  so  specialist  surface  warfare,  aviators  and  submariners  are  preferred. 
Although  DOT«feE  does  not  conduct  courses  for  new  personnel,  a  one  month 
handover  is  usually  required.  It  does,  however,  publish  a  comprehensive  staff 
orientation  guide  and  some  personnel  have  attended  the  OPTEVFOR  OTD  course. 
[Ref.  19] 

H.  USN  OT&E  IMPLEMENTATION  -  THE  DDG<51  PROGRAM 

The  objective  of  the  DDG-51  Program  is  to  build  the  next  class  of  destroyer  to 
replace  the  aging  ‘Coontz’  (DDG-37)  and  ‘Charles  F.  Adams’  (DDG-2)  classes.  The 
first  of  the  class,  the  USS  ARLEIGH  BURKE  has  recently  completed  OT-III.  It  is 
the  latest  USN  surface  warship  to  undergo  OT«feE,  thus  serving  as  the  most  recent 
example  of  the  implementation  of  USN  OT&E  policy  and  procedures.  The  DDG-51 
Program  is  under  the  management  of  the  AEGIS  Shipbuilding  Program  Manager 
(PMS  400)  and  is  sponsored  by  the  AEGIS  Program  Sponsor  (N865G).  PMS  400  is 
chartered  to  provide  comprehensive  direction  and  program  management  for  all 
aspects  of  system  development,  ship  acquisition  and  lifetime  support  preparation.  In 
this  task,  PMS  400  has  the  traditional  USN  responsibilities  of  a  ship  acquisition 
program  manager  and  also  bears  broadened  responsibilities  for  the  fleet  introduction 
and  lifetime  support  of  the  DDG-51  class  ships. [Ref.  33;p.  1] 

1.  Critical  Issues 

The  Critical  Operational  Issues  for  the  DDG-51,  as  defined  in  the  TEMP, 
are  subdivided  into  effectiveness  and  suitability  issues.  [Ref.  31:p.  rV-2]  The 
effectiveness  issues  for  DDG-51  are: 

•  Combat  System  Performance 

•  Mobility  System  Performance 

•  Command,  Control  and  Communications  Performance 


Support  System  Performance 

Survivability 

Tactics 


The  suitability  issues  considered  critical  for  DDG-51are: 

•  Reliability 

•  Maintainability 

•  Availability 

•  Lx)gistic  Supportability 

•  Compatibility 

•  Interoperability 

•  Training 

•  Human  Factors 

•  Safety 

•  Documentation 

•  Software  Supportability 

2.  DDG-51  OT&E  Program 

These  COIs  were  assessed  in  a  multi-phased  OT&E  program  as  shown  in 
the  schedule  in  Figure  8. 

a.  Initial  OT&E 

OT-II  was  split  into  two  major  system  areas,  Propulsion  System  and 
Combat  System  and  was  conducted  at  Land  Based  Test  Sites. 

(1)  OT-IIA  Propulsion  System  Testing  was  conducted  on  the  DDG-51 
Propulsion  System  at  the  Gas  Turbine  Ship  Land-Based  Engineering  Site 
(GTSLBES),  Naval  Ship  Systems  Engineering  Station  (NAVSSES),  Philadelphia, 
Pennsylvania.  It  is  an  Engineering  Development  Model  that  includes  the  Machineiy 
Control  System,  Data  Multiplex  System  and  the  engines,  reduction  gears,  thrust 


60 


bearings  and  propulsion-related  auxiliaries  of  the  complete  DDG-51  system.  A 
water  brake  was  used  to  simulate  the  propeller.  The  GTSLBS  was  exercised  in  a 
simulation  of  the  operational  environment  of  DDG-51class  ships.  Casualty  control 
drills,  maintenance  demonstrations,  and  individual  ship  propulsion  events  were 
conducted  during  424  hours  of  test  operations.These  included  full  power  runs  and 
emergency  crash  astern  maneuvers.  The  system  was  assessed  to  be  potentially 
operationally  effective  and  potentially  operationally  suitable.  [Ref  31:p.  IV-5] 


Figure  8:  DDG>51  Integrated  Test  Program  Schedule  [Ref.  31] 

(1)  OT-IIB  Combat  System  Testing  was  conducted  at  the  Combat 
System  Engineering  Development  Site  (CSEDS)  Moorestown,  New  Jersey. 
Following  a  philosophy  of  build  a  little,  test  a  little,  OT-IIB  was  conducted  in  three 
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phases.  OT-IIB-l  was  conducted  in  Jun  1986  on  a  partial  configuration  in  an  early 
stage  of  development  and  integration  to  support  the  program  Milestone  IIIA 
decision.  The  capability  in  all  mission  areas  could  not  be  evaluated  with  only  search, 
detection  and  tracking  of  AAW  targets  conducted.  No  engagements  of  air  targets 
were  conducted.  Twelve  raids  of  multiple  aircraft  were  conducted,  of  which  eleven 
were  in  an  ECM  environment,  including  various  jamming  modes  and  chaff. 
Dynamic  Test  Targets  (DTT)  were  also  introduced  to  evaluat?  system  performance. 
COMOPTEVFOR  concluded  that  the  combat  system  demonstrated  the  capability  to 
control,  integrate,  and  display  information  required  for  tactical  decisions  on  AAW 
targets  and  recommended  approval  of  the  DDG-51  Combat  System  for  limited  fleet 
introduction.  [Ref.  31:p.  IV-4] 

(2)  The  second  phase  (OT-IIB-2)  was  conducted  in  September  1988 
when  the  CSEDS  represented  an  operational  prototype  replicating  planned  human 
factors,  sensors,  information  processing  and  weapons  system  installations  in  the  real 
DDG-51.  Assessment  of  capability  in  all  mission  areas  was  conducted.  Thirty-four 
raids  of  manned  aircraft  were  conducted,  of  which  two  only  were  in  a  clear 
environment.  Additionally  DTTs  were  also  introduced  to  represent  antiship  missiles 
and  surface  hostile  threats.  Threat  submarine  evaluations,  both  active  and  passive 
were  conducted  as  were  Link  1 1  operations  with  aircraft,  surface  ships  and  shore 
stations.  Non  firing  engagements  with  both  Harpoon  and  Tomahawk  were  also 
conducted.  Operational  Suitability  data  were  collected  during  the  period.  On  test 
completion,  COMOPTEVFOR  recommended  continued  development  of  the 
Combat  System  and  the  conduct  of  an  additional  phase  of  OT-II  at  CSEDS  prior  to 
sea  trials. 

(3)  The  third  phase  (OT-IIB-3)  was  conducted  in  August  1990 
following  further  development  of  the  Combat  System.  The  testing  was 
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comprehensive  of  the  total  Combat  System’s  capabilities.  COMOPTEVFOR  noted 
that  performance  was  notably  improved  since  the  conduct  of  OT-IIB.  They  assessed 
the  system  as  being  potentially  operationally  effective  and  potentially  operationally 
suitable,  and  recommended  continued  development  of  the  Combat  System.  [Ref. 
31:p.  IV-6] 

OT-IIB-2  and  OT-IIB-3  were  conducted  to  support  continued  limited 
production.  DDG-51  crew  members  participated  in  the  OT-II  testing,  both  at  the 
GTLBES  at  NAVSSES  and  in  CSEDS  testing. 

b.  Follow  On  OT&E 

(1)  OT-IIIA  was  conducted  for  a  28  day  period  in  eariy  1992  on  board 
USS  ARLEIGH  BURKE.  Air,  surface  and  subsurface  targets  were  presented  in 
single  and  multithreat  environments.  Simulated  firings  against  manned  aircraft  and 
live  firings  against  threat-representative  targets  in  operationally  realistic  scenarios 
were  conducted  in  order  to  evaluate  the  performance  of  the  integrated  combat 
system  and  the  capability  of  the  mobility  and  support  systems  to  support  the  ship’s 
mission.  The  performance  of  all  the  ship  systems  designed  for  operation  with  other 
Navy  units  was  also  evaluated.  Suitability  tests,  off-ship  support  and  maintainability 
demonstrations  were  also  conducted.  [Ref.  31:p.  IV-9] 

(2)  OT-IIIB  is  planned  to  be  conducted  in  a  DDG-51  class  ship  to 
demonstrate  correction  of  deficiencies  identified  during  OT-IIIA  and  to  include 
previously  untested  systems. 

(3)  OT-IV  will  be  conducted  in  a  DDG-51  class  ship  to  demonstrate 
correction  of  deficiencies,  to  complete  deferred  or  incomplete  OT&E,  and  to 
evaluate  major  computer  software  revisions  as  well  as  the  DDG-51  class 
performance  in  battle  group  operations. 


c.  Timing  of  OT-m 

Table  6  shows  the  Post  Delivery  Schedule  of  DIX}-51.  Of  interest  is  the 
timing  of  OT-III  within  the  ship’s  program.  Note  that  OT-III  is  scheduled 
immediately  after  DT-III  testing  and  before  the  Workup  (REFTRA  and  FLEETEX). 
This  has  the  disadvantage  of  OT&E  not  being  conducted  on  a  fully  worked  up  ship 
with  the  crew  still  Developmental  Testing  oriented.  This  may  have  prejudiced  the 
OT&E  results. 


TABLE  6:  USS  ARLEIGH  BURKE  -  POST  DELIVERY  SCHEDULE  [Ref.  33:p.  5] 


Time 

Activity 

D  +  1  month 

Mobile  Training  Team  /  Training  Readiness  Evaluation 

D  +  2  m 

Light  Off  Examination  /  Fast  Cruise  /  Commissioning  /  Port  Visit 

D  +  3-4m 

Combat  Systems  Ship’ s  Qualification  Trials  (CSSQT) 

D  +  5m 

DT-III  (Demonstrate  ship  performs  as  designed) 

D  +  6m 

Final  Contract  Trials  /  OT-III 

D  +  7m 

OT-III  /  Operational  Propulsion  Plant  Examination 

D+8-lOm 

Post  Shakedown  Availability  (PSA) 

D+  11  m 

Refresher  Training  (REfTRA) 

D+  12  m 

Fleet  Exercise  (FLEETEX) 

D+  13  m 

Upkeep  (Maintenance  Period) 

D+  14  m 

OT-IV  (Backup  Events) 

D+  15m 

FT-EETEX  /  Battle  Ready 

D+  16  m 

Pre  Overseas  Movement  /  Deploy 

64 


3.  ‘Whole  Ship’ OT&E 

OT-III  conducted  on  the  DDG-51  was  unique  in  that  it  was  the  first  time  the 
USN  has  conducted  OT&E  on  a  whole  ship  in  a  multi-battle  situation,  in  a  fairly  free 
play  environment.  Previous  ship  OT&E,  e.g.,  on  the  “Aegis”  class  cruisers  (CG-47), 
was  essentially  a  combat  system  test  based  on  a  proven  hull,  where  the  rest  of  the 
ship  and  ship  integration  were  not  tested.  DOT&E  have  been  instrumental  in 
intensifying  efforts  to  create  an  operational  environment. 

a.  OT&E  Program  Development 

Following  the  guidance  contained  in  the  OTD  manual  [Ref.  15], 
OPTEVFOR  developed  a  test  design  matrix.  This  matrix  mapped  the  Top  Level 
Requirements  to  the  Schedule  of  Events  (SOE).  The  TLRs  were  general  and  related 
to  the  warfare  areas  of  AAW,  ASW,  ASUW  and  STW.  The  purpose  of  the  matrix 
was  to  ensure  that  all  TLRs  were  covered  by  an  operational  test  In  the  event  that  test 
resources  became  unavailable  or  a  system  casualty  occurred,  the  matrix  provided  a 
reference  of  what  had  been  accomplished,  what  remained  to  be  accomplished,  and 
the  feasibility  of  completing  the  test  objectives  if  OT  was  to  continue.  In  the 
evaluation  period,  the  “as-run  SOE”  was  laid  out  chronologically,  identifying  events 
by  mission  areas.  By  examining  the  as-run  table,  individual  mission  areas  were 
identified  for  analysis  and  periods  of  multiwarfare  could  be  identified  to  look  at 
synergistic  effects.[Ref.  21] 

h.  Integration  of  Engineering  and  Combat  System  Events 

Whole  ship  testing  includes  engineering  and  technical  problems 
combined  with  combat  system  activity.  The  OPTEVFOR  Operational  Test  Director 
controls  the  program  and  controls  injection  of  incidents.  A  balance  must  be  achieved 


between  engineering  incidents  and  related  degradation  of  the  combat  system  to 
ensure  the  overall  aim  of  the  test  is  achieved. 

c.  Operational  Realism 

The  OT&E  emphasised  operational  realism  by  testing  the  ship  as 
though  in  war,  with  a  full  political  background  to  set  tone  and  flavor.  The  situation 
consisted  of  an  overall  28  day  mission  consisting  of  individual,  but  interlinked, 
scenarios.  The  scenarios  were  developed  to  examine  the  ship’s  capability  to  operate 
as  a  fleet  asset  and  to  stress  the  installed  combat  systems’  capabilities  against  the 
threat.  They  were  designed  to  provide  a  realistic  threat  in  an  operational 
environment  and  an  element  of  surprise  to  provide  realistic  tests  to  support 
completion  of  the  OT«feE  objectives  and  resolutions  of  the  COIs.  The  Commanding 
Officer,  Combat  Information  Centre  team  and  the  ship’s  crew  were  allowed  to 
respond  to  the  tactical  situation  as  they  perceived  it.  [Ref.  31:p.  IV -9]  To  enforce 
realism,  operational  message  traffic  flow  at  the  same  rate  that  could  be  expected  in 
a  conflict  was  maintained  and  Damage  Control  incidents  were  introduced. 

d.  Limitations  to  Scope  of  Testing 

Despite  the  efforts  for  realism,  certain  practical  considerations,  time 
restrictions  and  resource  constraints  limited  the  scope  of  operational  testing.  These 
were  [Ref,  31;p.  IV- 10]: 

•  Targets  and  ECM  did  not  fully  replicate  the  threat  in  numbers  and 
characteristics  thus  precluding  fully  threat  representative  raids. 

•  The  CIWS,  ESM  and  ECM  were  not  tested  using  drone  targets.  Manned 
aircraft  were  used  for  ESM  and  ECM  tests  and  a  towed  target  was  used  for 
CIWS  firings. 

•  The  number  of  missiles  available  precluded  reengagements  and  multiple 
salvo  engagements  in  some  scenarios. 

•  The  geographical  location  of  test  ranges  and  their  facility  and  equipment 


66 


limitations  precluded  fully  threat  representative  target  presentations,  target 
density  and  open  ocean  geometries. 

•  The  duration  of  the  operational  testing  precluded  determination  of  reliability 
and  other  suitable  parameters,  forcing  the  use  of  development  testing  data 
where  credible  and  applicable. 

•  Operations  were  not  conducted  in  all  weather,  temperature,  humidity,  and 
wind  and  sea  conditions. 

€.  DDG-51  OT&E  Analysis 

The  OT«&E  of  DDG-51  highlighted  a  number  of  overall 
considerations.  [Ref.  21] 

(1)  Interfaces.  Interfaces  are  important  to  consider  in  whole  ship 
testing,  particularly  individual  systems  with  their  own  TEMPS  and  test  plans.  Whole 
ship  OT&E  looks  at  whole  mission  system  and  ship  performance  e.g.,  not  Just  sonar 
performance  but  ASW  system  performance.  Problems  occur  with  concurrent  sub¬ 
system  and  whole  ship  testing  due  to  different  critical  issues  and  objectives  of 
testing  which  create  conflicts  during  testing.  One  overall  test  plan  resolving  these 
issues  is  required. 

(2)  Statistical  Significance.  Some  tests,  e.g..  Tomahawk  firing,  can 
only  be  done  once.  To  achieve  results  of  statistical  significance,  a  number  of 
simulation  runs  were  conducted  using  new  and  varied  scenarios.  The  one  firing  was 
then  used  to  verify  the  simulation.  This  demonstrates  the  use  of  simulation  to  extend 
the  test  results. 

(3)  Qualitative  vs.  Quantitative.  Effort  was  made  throughout  DDG-51 
OT&E  to  use  quantitative  thresholds  based  on  quantitative  numbers,  whereas, 
previous  OT&E  of  the  AEGIS  class  cruisers  (CG-47)  qualitatively  assessed  all 
COIs. 
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(4)  Operationally  Realistic  Scenarios.  To  improve  the  measurement  of 
AAW  mission  performance  against  a  threat  representative  target  in  a  threat 
operational  scenario,  particularly  in  the  light  of  the  “Iranian  Airbus”  issue,  AAW 
scenarios  now  do  not  necessarily  include  everything  that  flies  is  hostile.  To  add  to 
the  operational  flavor,  a  political  situation  and  Rules  of  Engagement  (ROE)  are  used 
during  OT&E.  These  ROEs  change  due  to  the  political  situation  or  opposing  force 
actions  e.g.,  switching  on  fire  control  radar.  This  level  of  testing  adds  to  the  analysis 
problem  e.g.,  having  to  cross  the  Electronic  Warfare  detection  of  a  Fire  Control 
radar  to  the  aircraft,  with  the  subsequent  need  for  overall  exercise  analysis. 

(5)  Crew  Stress  under  Battle  Conditions.  The  DDG-51  OT&E 
program  include  the  measurement  of  crew  stress  under  battle  conditions,  both 
during  the  land  based  and  at  sea  testing.  They  observed  that  the  land  based  level  of 
operator  performance  was  significantly  better  than  at  sea.  The  USN  is  continuing  to 
pursue  methods  to  take  measurements  to  determine  how  much  of  the  performance 
degradation  from  land  based  testing  to  at  sea  testing  is  a  result  of  the  personnel.  The 
man  machine  interface,  the  concept  of  human  factors  and  operational  stress  are  a 
great  contributor  to  overall  system  performance.  One  objective  of  the  OT&E 
program  was  to  keep  the  crew  busy  during  the  test  period. 


(6)  System  Problems  vs.  People  Problems.  One  aim  of  OT&E  is  to 
assess  the  human  factors  relating  to  system  performance  which  involves  assessing 
the  adequacy  of  training,  ease  of  use,  etc.  It  does  not  include  the  performance 
assessment  of  an  individual.  This  can  sometimes  be  difficult  when  people  form  a 
vital  part  of  the  system.  OPTEVFOR  stress  they  are  not  drilling  the  crew,  but 
demonstrate  how  the  system  (which  may  include  people  in  the  loop)  responds.  It  is. 
therefore,  important  for  the  crew  to  be  ‘worked  up”  to  normal  fleet  standards  prior 


(7)  Independent  Third  Party  Data.  OT&E  needs  to  be  conducted  on  a 
range  or  with  another  ship/aircraft  to  get  independent  third  party  data  to  be  able  to 
reconstruct  the  evolution.  The  philosophy  of  testing  is  that  the  item  under  test  is 
immediately  suspect  so  if  data  from  this  system  is  required,  then  it  needs  to  be 
verified.  It  may  be  sample  verified  so  that  100%  duplication  is  not  required. 

(8)  Program  Manager  Perception.  Although  many  program  managers 
may  see  OT«feE  as  an  annoyance  and  unnecessary  burden  to  their  program,  PMS  400 
sees  OT&E  as  an  independent  assessment  to  demonstrate  that  its  on  the  right  track 
and  doing  the  right  thing.  [Ref.  22] 


L  U.S.  DEPARTMENT  OF  DEFENCE  OT&E  ISSUES 

U.S.  DoD  authorities  involved  in  ship  OT&E  see  a  number  of  macroscopic 
issues  relating  to  OT&E  in  the  USN  today. 

1.  Impact  of  Budget  Cutbacks 

a.  Acquisition  Strategy 

As  a  result  of  budget  cutbacks,  the  Secretary  of  Defence  is  developing 
a  new  acquisition  policy  for  systems  that  don’t  go  into  a  long  protracted  production 
phase.  A  prototyping  acquisition  strategy  may  be  implemented  where  one  of  kind  is 
built  and  the  technology  is  either  put  on  the  shelf,  or  into  limited  production  to  keep 
defence  industry  active.  More  emphasis  may  be  placed  on  R&D  than  on  production. 
Questions  arise  as  to  how  OT&E  fits  into  an  acquisition  strategy  that  does  not  lead 
to  production,  and  should  more  reliance  now  be  placed  on  DT&E  and  less  on 
OT&E.  [Ref.  19]  Current  thinking  is  that  more  importance  may  be  placed  on 
modeling  and  simulation  to  estimate  operational  effectiveness  and  operational 


suitability  of  advanced  technical  demonstrators,  to  minimise  the  operational  risk  if 
production  and  deployment  is  required  in  an  emergency. 

b.  OT&E  Conduct 

Questions  arise  also  of  how  OT&E  is  to  be  conducted  in  a  reduced 
budget  environment.  With  reduced  production  runs,  the  relative  cost  of  OT&E  in 
relation  to  overall  program  cost  will  increase.  This  may  not  be  seen  as  acceptable 
and  pressure  may  be  applied  to  reduce  the  absolute  cost  of  OT&E.  The  breadth  of 
assessment  may  be  diminished  using  less  test  assets.  There  may  be  also  a  possibility 
of  a  central  OT&E  agency.  With  DOT&E  becoming  an  increasingly  aggre'^sive 
participant  in  OT&E,  testing  in  the  future  could  be  directed  exclusively  by 
DOT&E  [Ref.  19] 

c.  Increasing  Modeling  and  Simulation 

Modeling  and  simulation  (M&S)  capabilities  are  going  to  play  a  larger 
role  in  contributing  to  the  OT  process.  Although  M&S  will  never  replace  operational 
testing,  it  will  be  able  to  supplement  actual  testing  to  a  greater  degree  in  certain 
areas.  Technological  advances  in  M&S  may  make  these  tools  now  more  of  a 
candidate  to  be  used  for  OT&E.  Typically  M&S  is  more  of  a  study  or  analysis  or 
developmental  type  tool,  but  are  rapidly  developing  capabilities  that  will  offer 
advantages  to  the  OT  process.[Ref.  19] [Ref.  21] 

2.  Operational  Realism 

Following  GAO  criticism  of  lack  of  realism  [Ref.  36]  and  the  ‘whole  ship’ 
01  &E  concept,  test  scenarios  can  be  expected  to  be  more  realistic  in  the  future.  To 
obtain  the  maximum  benefit  from  operational  tests,  they  will  also  need  to  be  more 
quantitative.  This  may  include  the  development  of  integrated  multiwaifare  ranges  to 
support  whole  ship  testing.[Ref.  19][Ref.  21] 
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J.  USNOT&E  SUMMARY 

1 .  Importance  of  OT&  E 

OT&E  is  the  “final  exam”  in  the  U.S.  DoD  acquisition  process,  an 
empirical  method  of  ensuring  that  sufficient  technical  return  is  made  on  acquisition 
investments  and  that  a  new  system  is  fully  capable  of  meeting  the  Fleet’s  needs  prior 
to  production  [Ref.  12].  As  a  result,  the  U.S.  place  high  importance  on  OT&E.  The 
requirement  for  the  conduct  of  OT&E  is  mandated  by  Congress  and  incorporated  in 
the  laws  of  the  U.S.  OT&E  is  viewed  as  being  more  important  as  a  basis  for  a 
decision  to  proceed  beyond  LRIP,  than  on  the  introduction  into  service  of  the  final 
production  item.  This  is  because  the  “big  bucks”  of  most  Defence  acquisitions  are 
spent  during  the  production  phase.  OT&E,  however,  continues  to  play  an  important 
role  in  the  assessment  of  systems  for  Fleet  Introduction  and  has  increased  its 
importance  in  the  assessment  of  complete  ships  for  Fleet  Introduction. 

2.  Need  for  OT&E 

The  USN  need  for  OT&E  stems  from  the  almost  exclusive  reliance  by  the 
U.S.  on  indigenous  weapons  development  and  production.  They  have  a  history  of 
large,  risky,  developmental  programs  which  push  the  state  of  the  art  in  weapons 
technology.  The  outputs  of  these  programs  require  the  assessment  of  operational 
effectiveness  and  suitability  before  being  committed  to  production  and  subsequent 
introduction  into  the  fleet.  The  U.S.  has  been  involved  in  a  number  of  military 
conflicts  which  demonstrate  the  need  for  operationally  effective  and  suitable 
systems.  The  lessons  learned  from  Vietnam,  and  Gulf  operations  including  the  USS 
STARK  and  USS  VINCENNES  incidents,  highlighted  problems  which  DT&E 
alone  may  not  identify. 
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3.  OT&E  Agency  Independence 

Independence  is  the  key  to  the  effectiveness  of  OT&E  in  the  U.S. 
acquisition  process.  OPTEVFOR  is  the  sole  independent  OT&E  agency  within  the 
USN,  reporting  directly  to  CNO,  and  DOT&E  provides  independent  oversight  and 
coordination  of  the  military  services’  planning  and  execution  of  OT&E,  reporting 
directly  to  the  Secretary  of  Defence.  OT&E  agency  independence  is  designed  to 
ensure  impartiality,  honest  and  open  reporting,  with  a  minimum  of  political 
interference. 

4.  T&E  Delineation 

The  USN  have  a  clear  delineation  between  the  different  types  of  T&E. 
OPTEVFOR  conduct  only  OT&E,  and  OT&E  within  the  USN  is  conducted  solely 
by  OPTEVFOR.  Thus,  there  appears  to  be  good  understanding  in  the  USN  of  what 
is,  and  what  isn’t,  OT&E. 

5.  Whole  Ship  OT&E 

Whole  ship  OT&E  is  conducted  in  a  dedicated  period  following  CSSQT 
and  Final  Contract  Trials,  and  consists  of  fully  assessed  free  play,  multi -threat 
scenarios.  The  whole  ship  is  viewed  as  an  integrated  warfare  system,  rather  than  as 
a  platform  for  warfare  systems. 

6.  Clear  Guidance 

The  requirement  for  OT&E,  and  its  organisation  and  methodology  are  well 
documented  within  the  USN,  providing  clear  guidance  to  acquisition  and  operations 
personnel.  Also,  training  in  OT&E  is  provided,  further  improving  the  capability  and 
understanding  of  personnel  involved  with  OT&E. 
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7.  Summary 

The  USN  OT&E  system  has,  by  necessity,  developed  into  a  comparatively 
well  organised,  well  documented  and  effective,  if  complex,  system  that  meets  the 
requirements  of  the  U.S.  Defence  acquisition  process. 
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IV.  RAN  SHIP  OPERATIONAL  TEST  &  EVALUATION 


The  recent  disestablishment  of  the  RAM’s  independent  Operational  Test  and 
Evaluation  authority,  the  RAN  Trials  and  Assessing  Unit  (RANTAU),  and  its 
subsequent  integration  with  Maritime  Headquarters  (MHQ),  marks  a  continuing 
shift  in  RAN  OT&E  policy.  This  chapter  describes  the  RAN  OT&E  system,  its 
relationship  with  the  other  categories  of  T&E  and  its  role  in  the  RAN  ship 
acceptance  process.  As  RAN  T«feE  documentation  does  not  yet  reflect  the 
disbanding  of  RANTAU,  the  OT&E  system  is  described  as  it  was  prior  to  May 
1992,  and  then  the  new  organisation  will  be  introduced. 

A-  AUSTRALIAN  DEFENCE  ACQUISITION  PROCESS 

Acquisition  of  major  capital  equipment  for  the  Australian  Defence  Force  ( ADF) 
is  managed  by  the  Capital  Procurement  Organisation  (CPO)  within  the  Department 
of  Defence.  The  defence  system  life  cycle  consists  of  the  following  phases; 

•  Conceptual 

•  Demonstration  and  Evaluation 

•  Full  Scale  Engineering  Development 

•  Production 

•  In  Service 

Similar,  but  not  identical,  to  the  U.S,  system,  these  phases  are  generally 
separated  by  the  following  milestones: 

•  Program  Initiation 

•  Full  Scale  Engineering  Development 

•  Production  Approval 
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•  Preliminary  Acceptance  (PA)/  Delivery 

•  Acceptance  into  Naval  Service  (AIMS) 

•  Operational  Acceptance  (OA) 


FULL SCALE 

PROGRAM  ENGINEERING  PRODUCTION  PRELIM  *INS  OA 

INITIATION  DEVELOPMENT  APPROVAL  ACCEPT 

APPROVAL 


Figure  9:  Australian  Acquisition  Process  for  Major  Programs 


1.  Development  Vs.  Procurement 

The  completion  of  all  acquisition  phases  applies  only  to  indigenously 
developed  systems.  Systems  procured  overseas  generally  undergo  only  those  phases 
from  Milestone  3  onwards.  Due  to  the  RAN’s  small  size  and  limited  requirements, 
it  is  not  practical  or  achievable  to  develop  all  required  systems  locally.  In  many 
cases,  systems  with  the  fundamental  capabilities  required  by  the  ADF  are  available 
overseas,  however,  some  of  these  systems  require  additional  development, 
adaptation  or  integration  to  meet  Australian  specific  requirements.  Many  of  these 
systems  require  only  minimal  adaptation  (e.g.,  in  their  communications  fit),  while 
others  are  more  extensive  (e.g.,  ANZAC  class  frigates  and  COLLINS  class 
submarines).  This  leads  to  different  types  of  system  acquisition  programs. 
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2.  Types  of  Acquisitions 

The  following  are  the  principal  ways  Australia  acquires  defence 
capabilities  [Ref.  6:p.  86]: 

a.  Indigenous  Design  and  Development 

Local  research  and  development  can  be  undertaken  to  meet  special 
local  requirements,  or  as  consequence  of  research  undertaken  to  maintain  the 
technology  base.  The  new  system  is  subject  to  all  phases  of  the  acquisition  process. 

b.  Local  Development  and  Production 

Naval  Adaptation  programs  in  which  an  overseas  design  is  purchased, 
but  then  undergoes  modifications  to  meet  specific  RAN  requirements.  Some 
engineering  development  work  may  be  needed. 

c.  Local  Production  of  Overseas  Designs 

In  some  cases,  the  requirements  of  the  ADF  can  be  met  most  cheaply 
and  expeditiously  by  existing  overseas  systems  where  there  is  no  economic  or 
technical  prospect  of  a  local  competitor.  The  system  may  be  built  locally,  where  it 
would  be  subject  to  Production  A  pproval. 

d.  Import  of  Overseas  Equipment 

Non  development  programs,  where  systems  are  purchased  “off-the- 
shelf’  from  a  foreign  source,  without  modification,  are  subject  to  the  acceptance 
milestones  only. 

Each  of  these  types  of  acquisition  differ  in  their  technical  and  operational 
risk  and  hence  T&E  requirements.  In  general,  Australia  has  a  policy  of  buying  low 
risk,  operationally  proven  systems,  which  may  be  modified  to  meet  local 
requirements. 


R  AUSTRALIAN  DEFENCE  T&E  DEHNITIONS 


The  categories  of  ship  T&E  used  in  the  Australian  E)oD  have  been  derived  from 
the  U.S.  DoD  instructions,  but  interpreted  to  accord  with  Australian  requirements 
[Ref.  38:p.  14-1],  There  is  a  hierarchy  of  documents  pertaining  to  T&E  within  the 
DoD  and  RAN. 

1.  DoD  T&E  Deflnitions 

At  the  DoD  level,  the  Australian  Capital  Equipment  Procurement  Manual 
(CEPMAN  1)  [Ref.  38]  defines  the  types  of  T&E  applicable  to  the  Australian  DoD 
as  follows: 

a.  Development  Test  and  Evaluation  (DT&E) 

DT&E  assists  the  development  specification,  design  and  procurement 
process  and  to  verify  the  attainment  of  development  specifications. 

b.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  is  to  assist  the  development  of  a  procurement  specification  and 

to  verify  compliance  with  its  requirements.  It  is  generally  associated  with  a 
production  activity  and  normally  forms  the  basis  of  qualified  acceptance  of  the 
equipment. 

c.  Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  to  assist  the  development  of  an  operational  capability 
specification  and  to  measure  compliance  with  its  requirements.  CEPMAN  1  also 
defines  a  sub-grouping  within  the  OT&E  classification  -  the  Operational  Evaluation 
(OPEVAL).  An  OPEVAL  covers  tests  and  evaluation  on  production  representative 
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baseline  equipment  using  the  maintenance  and  support  personnel  and  equipment  for 
normal  operational  use  and  aims  to: 

•  Demonstrate  operational  effectiveness  and  suitability 

•  Provide  data  to  assist  in  the  development  of  tactical  aspects  of  the  equipment 

•  Verify  data,  handbooks  and  documentation  covering  the  operation  of  the 
system. 

2.  RAN  T&E  Definitions 

These  DoD  definitions  are  further  refined  by  the  RAN.  The  Defence 
Instruction  for  RAN  Test  and  Evaluation  Policy  [Ref.  44]  defines  the  types  of  T&E 
as  follows: 

CL  Development  Test  and  E valuation  (DT &E) 

DT&E  denotes  that  T&E  conducted  in  order  to  demonstrate  the 
progressive  achievement  of  design  requirements  and  to  show  that  the  article  will 
meet  its  technical  specifications.  DT&E  is  divided  into  three  phases  as  shown  in 
Table  7. 


TABLE  7:  PHASES  OF  DT&E  [Ref.  44:p.  2] 


T&E  Type 

DT-l 


Description 


Validation  of  design  concept 


DT-2 


Proving  design 


DT-3 


Demonstration  that  production  meets  required  technical 
characteristics 
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b.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

PAT&E  denotes  that  T&E  conducted  by  or  on  behalf  of  the 
procurement  agency  to  determine  whether  contracted  provisions  have  been 
satisfied.  PAT&E  is  divided  into  two  sections  with  a  total  of  eight  phases  as  shown 
in  Table  8. 


TABLE  8:  PHASES  OF  PAT&E  [Ref.  44:p.  2] 


PAT&E  Part 

T&E  Type 

Description 

Part  1 

PAT-0 

Design  and  engineering  development  tests 

PAT-1 

Production  and  bum-in  tests 

PAT-2 

Environmental  qualification  tests 

PAT-3 

System  development  tests 

PAT4 

Harbour  testing 

PAT-5 

Sea  testing 

Part  2 

PAT-^ 

Operational  and  Qualification  Trials 
(conducted  with  OT-3) 

PAT-7 

Follow  on  Operational  T&E 
(conducted  with  OT-4) 

c.  Operational  Test  and  Evaluation  ( OT &E) 

OT&E  is  that  T&E  conducted  to  estimate  the  operational  effectiveness 
and  suitability  by  an  authority  independent  of  the  development  and  production 
agencies.  OT&E  is  divided  into  five  phases  as  shown  in  Table  9.  Of  note  is  that 
Certification/Qualification  trials  are  included  in  OT-3.  These  trials  consist  of 
Communications,  Emitter  and  other  individual  system  certifications.  These  trials 
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also  include  a  five  to  seven  week  Combat  System  Ship  Qualification  Trial  (CSSQT) 
that: 


•  Demonstrates  the  maintainability  and  operability  of  those  equipments/ 
systems  that  are  included  in  the  CSSQT  program  through  accomplishment  of 
equipment/system  Planned  Maintenance  System  (PMS),  aircraft/balloon 
tracking  exercises,  practice  missile  firings  and  culminating  in  live  firing 
exercises. 

•  Provides  training  and  familiarisation  to  ship  personnel  in  the  maintenance 
and  operation  of  the  installed  equipments/systems  that  are  included  in  the 
CSSQT  program. 

•  Identifies  design  problems  in  the  equipment/system  installed. 

•  Identify  any  deficiencies  that  may  exist  in  the  CSSQT  support  elements,  i.e., 
documentation,  logistics,  test  equipment  or  training. 


TABLE  9:  PHASES  OF  OT&E  [Ref.  44:p.  2] 


T&Eiype 

Description 

OT-l 

Operational  assessment  of  the  development  proposal 

ar-2 

Demonstration  of  achievement  of  program  requirements  for 
operational  effectiveness  and  suitability  of  a  prototype 
(OPEVAL)  to  support  proceeding  to  full  production 

ar-3 

Demonstration  of  achievement  of  program  requirements  for 
operational  effectiveness  and  suitability  on  production  of  ship/ 
aircraft/system,  normally  in  independent  operations  using  normal 
Fleet  personnel.  Certificatioii/Qualification  IVials  include 
limited  reliability,  maintainability,  availability  and  logistic 
supportability  assessments. 

CfT^ 

Demonstration  of  achievement  of  program  requirements  for 
operational  effectiveness  and  operational  suitability  on 
production  of  ship/aircraft/system  using  normal  Fleet  personnel 
in  a  multi-force,  multi-threat  environment.  Includes  detailed 
reliability,  maintainability,  availability  and  logistic  supportability 
assessments. 

cfr-5 

Follow-on  OT  for  assessment  after  modernisations,  new 
applications  or  defect  rectifications  after  OT-4 
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C  T&E  IN  THE  ACQUISITION  PROCESS 

1.  The  Need  for  T&E 

The  most  recent  Australian  Government  “White  Paper”  on  defence  policy 
[Ref.  6:p.  70]  states: 

We  need  to  be  able  to  determine  the  performance  in  our  own  environment  of 
equipment  of  both  overseas  and  local  origin  and  to  modify  and  adapt  overseas 
equipment  as  necessary  to  improve  its  performance  in  our  likely  theatres  of 
military  operations. 

Also,  the  guidance  for  Australian  defence  acquisition  [Ref.  38;p.  14-1] 
recognises  that: 

The  evaluation  of  military  weapons  systems  and  their  individual  component 
systems  is  essential  to  this  policy.  Analysis  of  the  results  of  testing  performed 
as  part  of  the  Test  and  Evaluation  (T&E)  of  a  capital  equipment  project  may 
assist  in  the  identification  of  problem  areas,  allow  timely  corrective  action  to 
be  taken,  and  reduce  the  element  of  risk  in  major  decisions. 

So  the  need  for  local  T&E  is  recognised  by  the  Australian  Government  and 
the  DoD,  for  both  indigenous  and  overseas  systems.  The  responsibility  for  T&E  of 
a  system  rests  with  organisations  with  responsibility  for  the  design  approval, 
certification  or  procurement  of  equipment,  who  have  the  authority  and  responsibility 
to  conduct  (or  require  the  conduct  of)  T&E.  All  project  requirements  are  coordinated 
by  the  Project  Manager,  who  considers  whether  or  not  T&E  can  be  conducted  within 
available  project  resources. [Ref.  38:p.  14-2]  However,  the  importance  of  OT&E  in 
particular,  is  not  specifically  addressed  in  any  of  these  T&E  policy  documents. 

2.  OT&E  Contributions  at  Major  Milestones 

OT&E  in  the  RAN  may  be  conducted  during  each  phase  in  the  acquisition 
process,  but  it  is  not  essential  to  a  local  system  meeting  a  development  or  production 
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milestone,  or  the  local  production  or  purchase  decision  of  an  overseas  system.  It  is 
however,  required  before  a  system  achieves  Acceptance  into  Naval  Service  or 
Operational  Acceptance.  The  phasing  of  T&E  during  the  acquisition  process  is 
detailed  in  Figure  10. 


FULLSCALE 
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Figure  10:  Test  &  Evaluation  in  the  Acquisition  Process  [Ref.  44:p.  B>1] 


The  contribution  of  OT&E  to  the  acquisition  milestones  includes: 

a.  OT&E  prior  to  Program  Initiation 

No  role  is  documented  for  OT&E  in  the  Conceptual  Phase. 

b.  OT&E  prior  to  Engineering  Development  Approval 

During  the  E>emonstration  and  Evaluation  Phase,  an  operational 
assessment  of  the  development  proposal  (OT-1)  may  be  conducted.  The  results  of 
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which  may  contribute  to  the  decision  to  undertake  Full  Scale  Engineering 
Development. 

c.  OT &E  prior  to  Production  Approval 

Towards  the  end  of  the  Full  Scale  Engineering  Development  Phase,  a 
demonstration  of  achievement  of  program  requirements  for  operational 
effectiveness  (OT-2)  and  suitability  of  a  prototype  (OPEVAL)  to  support 
proceeding  to  full  production.  This  phase  may  include  system/software  assessment 
at  a  Land  Based  Test  Site  [Ref.  44:p.  2].  The  results  of  OT-2,  if  conducted,  may  be 
used  to  support  a  local  production  or  purchase  decision. 

d.  OT dcE  prior  to  Preliminary  Acceptance 

No  role  is  documented  for  OT&E  in  the  Production  Phase. 

e.  OT &.E  prior  to  Acceptance  into  Naval  Service 

Although  Figure  10,  indicates  an  OT-3  phase  prior  to  AINS,  no  specific 
requirement  is  documented  in  RAN  policy  for  the  conduct  of  OT-3  prior  to  AINS. 
The  prerequisites  for  AINS  detailed  in  Reference  41  p  15.1 1,  are  described  in  terms 
of  PAT&E  Part  2,  and  make  no  mention  of  OT&E.  At  the  working  level,  however, 
the  RANTAU  Standard  Operating  Procedure  for  OT&E  [Ref.  53]  includes  the 
conduct  of  OT-3  prior  to  AINS,  and  the  results  of  OT-3  have  usually  been 
considered  in  the  AINS  decision. 

f.  OT&E  prior  to  Operational  Acceptance 

For  first  of  class  ships  it  will  not  normally  be  possible  to  trial  and 
evaluate  all  aspects  of  the  operational  performance  and  maintainability  prior  to 
AINS.  Therefore,  the  first  of  class  usually  undergo  follow-on  OT&E  (FOT&E)  after 
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AINS,  leading  to  Operational  Acceptance  [Ref.  44:p.  A-1].  This  OT-4  Phase 
consists  of: 

•  Evaluating  the  ship  as  part  of  a  force  during  an  exercise,  in  order  to  develop 
standard  operating  procedures  and  tactics,  and  to  explore  the  limits  of 
capability,  etc. 

•  Evaluating  the  maintainability  and  stores  support  for  equipments  and  systems 
to  determine  whether  or  not  the  complement  t,  stores  allowances, 
documentation  are  adequate.  [Ref.  41:p.  15.14] 

g.  OT&E  Contributions  post  Operational  Acceptance 

OT-5  is  follow-on  OT&E  for  assessment  after  modernisations,  new 
applications  or  defect  rectifications  after  Cyr-4  [Ref.  44:p.  3]. 

The  OT&E  contributions  to  the  Acquisition  Milestones  are  by  no  means 
definitive.  There  is  no  documented  requirement  for  the  conduct  of  OT- 1  and/or  OT- 
2  prior  to  a  production  or  purchase  decision  of  major  acquisitions.  The  decision  to 
conduct  OT&E  being  the  responsibility  of  the  Sponsor,  Project  Manager  and  the 
Defence  Acquisition  Committees.  RAN  T&E  policy  documentation  almost 
exclusively  focuses  on  the  post  production  acceptance  phases.  Even  during  the 
acceptance  phase,  however,  the  requirement  for  OT-3  is  clouded  in  definition 
differences  between  policy  documents.  The  RAN  appears  to  have  its  focus  on 
Production  Acceptance  Testing,  possibly  because  most  systems  procured  in  the  past, 
were  already  in  service  with  a  large  parent  Navy.  Tactical  development  was  not 
necessary  because  the  tactics  also  were  acquired  with  the  system. 

3.  Production  Acceptance  Test  and  Evaluation 

As  defined  previously.  Production  Acceptance  Test  and  Evaluation 
(PAT&E)  is  the  testing  conducted  to  demonstrate  that  systems  meet  contract 
specifications  and  requirements:  also  that  items/systems  are  properly  installed  and 
operaUe  onboard  the  ship.  The  objective  PAT&E  is  to  confirm  that  the  contractor 
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has  met  contractual  obligations,  and  hence  the  Commonwealth  may  correctly  take 
delivery  of  the  ship.  [Ref.  41  :p.  14-2]  Much  emphasis  is  given  within  the  RAN  to 
PAT&E,  known  under  various  terms  including  Inspections,  Tests  and  Trials  (ITT), 
Harbour  Acceptance  Trials  (HATs)  and  Sea  Acceptance  Trials(SATs).  In  recent  years 
the  RAN  has  adopted  the  U.S.  Total  Ship  Test  Program  (TSTP)  concept,  initially 
with  the  modernisation  of  the  DDGs,  and  now  in  the  testing  of  the  locally  produced 
FFG-7  class,  under  the  Australian  Frigate  Project.  The  adoption  of  the  TSTP  by  the 
RAN  included  the  redefinition  of  the  test  phases.  Where  the  U.S.  system  included 
only  PAT&E,  the  RAN  has  included  elements  of  DT&E  and  OT&E,  in  addition  to 
PAT&E,  and  gave  them  PAT  designations. 

a.  Integrated  Test  Package 

Under  the  TSTP,  the  program  and  content  of  the  formal  ITT  required 
by  the  contract  is  usually  designated  the  Integrated  Test  Package  (FTP).  Although  in 
the  Australian  system,  Government  testing  is  also  included.  The  categories  of  ITT 
testing  are  listed  in  Table  10. 


TABLE  10;  CATEGORIES  OF  ITT  [Ref.  41  ;p.  14.3] 


PAT&E 

Category 

T&El>pe 

Description 

Part  1 

0 

PAT-0 

Design  and  engineering  development  tests 

1 

PAT-1 

Production  and  bum-in  tests 

2 

PAT-2 

Environmental  qualification  tests 

3 

PAT-3 

System  development  tests 

4 

PAT^ 

Harbour  testing 

5 

PAT-5 

Sea  testing 

Part  2 

6 

PAT-6 

Ship  Operational  and  Qualification  Trials 

7 

PAT-7 

Fdlow  on  Operational  T&E 
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b.  Overlap  of  PAT&E  and  OT&E 

PAT&E  Part  2  covers  the  period  from  Preliminary  Acceptance  to  the 
AINS  and  equates  to  PAT-6.  The  Project  Director  has  responsibility  for  the  program 
and  its  funding  and  RANTAU  is  responsible  for  the  detailed  schedule  of  events  and 
the  provision  of  test  and  trials  resources.  Testing  includes: 

•  Ship’s  staff  familiarization,  shakedown  and  safety  drills 

•  Installation  and  testing  of  specialised  equipment 

•  Tempest  test 

•  Harbour  trials  of  weapons,  communication  and  combat  data  systems,  etc. 

•  Ship  Qualification  Trials 

•  Degaussing  and  Noise  Ranging 

However,  some  of  these  testing  activities  are  also  included  as  part  of 
OT-3,  and  both  PAT-6  and  OT-3  are  conducted  by  the  same  authority  (RANTAU), 
during  the  same  period,  and  using  the  same  T&E  personnel.  Similarly,  the  PAT-7 
phase  equates  to  OT-4. 

D.  RAN  OT&E  ORGANISATION 

The  RAN  community  responsible  for  planning,  programming,  budgeting  and 
evaluating  OT&E  forms  an  integral  part  of  the  more  general  T&E  organisation. 
OT&E  in  the  RAN  is  not  treated  as  being  special  or  unique,  and  no  organisation 
deals  solely  with  OT&E  The  RAN  organisational  structure  has  changed  over  the 
past  few  years  which  has  impacted  on  the  structure  of  the  OT&E  community. 

1.  OT&E  Organisation  Elements 

The  functional  elements  of  an  OT&E  organisational  structure  include: 

•  Policy  Maker  and  Overseer 

•  Sponsor 


•  Developing  Agency 

•  Tester  and  Evaluator 

•  User 

•  Coordinator 

a.  Policy  Maker  and  Overseer. 

There  is  no  clear  OT&E  policy  maker  within  the  Australian  DoD  or  the 
RAN.  The  Chief  of  Capital  Procurement  within  DoD  has  a  role  in  the  formulation 
of  capital  equipment  procurement  policy  for  the  guidance  of  Service  Materiel 
Divisions.  The  CPO  manual  [Ref.  38]  addresses  T&E,  but  more  in  the  form  of 
guidance  rather  than  policy.  The  Director  General  of  Naval  Warfare  (DGNW)  in  the 
£)eputy  Chief  of  Naval  Staff  (DCNS)  Division,  drafted  the  RAN  Test  and  Evaluation 
Policy  Defence  Instruction  [Ref.  44].  DGNW  however,  is  now  defunct  with  the 
result  that  Test  and  Evaluation  policy  for  Navy  is  presently  in  a  state  of  flux,  with 
no  office  having  assumed  the  responsibility  [Ref.  50]. 

b.  Sponsor 

The  sponsor  is  responsible  for  monitcaing  operational  requirements  and 
warfare  concepts,  the  development  of  concepts  for  operations  for  naval  warfare 
systems,  and  the  raising  of  staff  targets  and  requirements  as  required.  He/She  is  also 
responsible  for  the  subsequent  trials  and  continuing  overview  to  ensure  that  the 
equipment  meets  the  requirement.  Originally  the  responsibility  of  DGNW,  this 
function  for  major  acquisitions  was  assumed  by  the  Director  General  Force 
Development  (Sea)  in  Headquarters,  Australian  Defence  Force  (HQADF).  This 
centralised  force  development  and  user  requirements  organisation  produces  the 
Naval  Capability  Proposals  (NCP).  The  decision  to  subject  an  acquisition  to  T&E  is 
initially  decided  by  the  sponsor  and  detailed  in  the  capability  proposal  [Ref.  44:p.  1] 
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c.  Developing  Agency 

When  a  project  is  approved,  a  project  office  is  formed  within  the 
Materiel  Division  responsible  for  acquisition  of  the  capital  equipment  to  meet  the 
approved  NCP.  The  nature  and  extent  of  the  T&E  to  be  conducted  will  be  decided 
by  the  Project  Director  in  consultation  with  the  Design  Approval  Authority  and 
RANTAU  and  detailed  in  the  TEMP  [Ref.  44:p.  1].  The  Project  Manager,  in 
consultation  with  operational,  technical  and  maintenance  authorities,  is  to  fully 
investigate  the  necessity  for,  and  likely  scope  of.  Defence  T&E  [Ref.  38:p.  14-5]. 

d.  Operational  Tester  and  Evaluator 

The  RAN  Trials  and  Assessing  Unit  (RANTAU)  was  the  RAN’s 
OT&E  authority  with  a  mission  to  provide  an  independent  evaluation,  audit  and 
training  service  in  response  to  current  and  future  Maritime  Force  requirements. 
Located  in  North  Sydney,  near  the  main  Fleet  Base,  the  trials  related  functions  of 
RANTAU  included[Ref.  43:p.  1]: 

•  Conduct  of  operational  effectiveness  and  operational  suitability  trials  on 
ships  and  facilities  under  construction,  modernisation,  conversion  or 
extended  refit  in  support  of  AINS. 

•  Conduct  test  and  evaluations,  investigations  and  inspections  of  selected 
equipment  when  tasked. 

•  Audit  and  witness  selected  Harbour  Acceptance  Trials  (HATS),  Sea 
Acceptance  Trials  (SATS)  and  Test  Procedures  which  have  operational 
implications  for  ships  and  facilities  (now  known  as  PAT  4  and  5). 

•  Conduct  Ship  Qualification  Trials. 

RANTAU,  although  the  OT&E  authority,  also  conducted  PAT&E  on 
behalf  of  Project  Directors  and/or  the  Maritime  Commander.  With  the 
disestablishment  of  RANTAU  in  May  1992,  the  trials  functions  were  transferred  to 
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the  Commander,  Test  and  Evaluation  (CTE)  under  the  Chief  Staff  Officer 
(Engineering)  in  the  Maritime  Command. 

e.  User 

The  Maritime  Commander,  and  the  individual  ships’  Commanding 
Officers  and  crews  are  the  ultimate  users  of  a  ship,  RAN  T&E  documentation, 
however,  does  not  address  their  involvement  in  T&E.  The  Commanding  Officer  of 
a  ship  under  test  usually  provides  RANTAU  with  his  opinions  on  the  ship’s 
performance  at  the  conclusion  of  a  trials  period. 

f.  Coordinator 

The  Director  of  Naval  User  Requirements  (DNUR),  under  DGNW, 
once  acted  as  the  Navy  Office  representative  for  RANTAU  within  Navy  Office, 
Canberra.  He  provided  a  coordination/liaison  role  between  RANTAU  and  the 
Materiel  Division,  by  attending  project  meetings  and  highlighting  T&E 
requirements. 

2.  Changes  to  the  RAN  OT&E  Organisation 

The  OT&E  organisation  within  the  RAN  has  changed  dramatically  over  the 
past  few  years.  Figure  1 1  illustrates  the  OT&E  organisation  in  1987.  The  Director, 
RANTAU  (of  Captain  rank)  was  directly  responsible  to  the  Assistant  Chief  of  Naval 
Staff  -  Development  (ACDEV-N).  Also  under  ACDEV-N,  the  Director  General 
Naval  Warfare  (DGNW)  sponsored  all  major  acquisitions,  and  gave  T&E  policy 
direction.  Under  DGNW,  the  Director  of  Naval  User  Requirements  (DNUR), 
provided  RANTAU  representation  and  performed  an  OT&E  coordination  role 
within  Navy  Office.  Thus  the  OT&E  authority  was  responsible  to  the  developer  of 
the  user  requirement,  and  was  independent  of  the  developing  and  production 
agencies. 
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Figure  11:  RAN  OT&E  Organisation  circa.  1987 


By  1990,  the  structure  outlined  in  Figure  12  had  developed.  With  the 
centralisation  of  military  policy  development  and  resource  planning,  the 
Development  Division  within  the  RAN  was  disbanded.  DGNW,  with  reduced  user 
requirement  responsibilities,  moved  under  the  Deputy  Chief  of  Naval  Staff  (DCNS), 
The  emerging  HQADF  now  became  the  Sponsors  for  major  acquisitions.  The 
Director  General  Force  Development  (Sea)  (DGFD(SEA))  having  particular 
responsibility  for  “Sea”  Capability  Proposals  [Ref.  54].  DGNW,  however,  retained 
responsibility  for  T&E  policy  within  the  RAN.  RANTAU  was  also  transferred  under 
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DCNS.  As  the  sponsorship  of  major  acquisitions  was  now  held  by  HQADF,  the 
OT&E  authority  was  now  no  longer  responsible  to  the  developer  of  the  user 
requirement.  DNUR  was  dissolved  and  the  OT&E  authority  lost  its  representation 
in  Navy  Office. 


Figure  12:  RAN  OT&E  Organisation  circa.  1990 


By  the  end  of  May  1992  (Figure  13),  RANTAU  was  disestablished  and  its 
trials  functions  were  transferred  to  the  Maritime  Commander  as  the  Commander 
Test  and  Evaluation  (CTE)  under  the  Chief  Staff  Officer  (Engineering)  (CSO(E)). 
DGNW  was  dissolved  !»nd  his  now  reduced  functions  were  assumed  by  the  Director 
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Naval  Warfare  (DNW)  under  the  newly  created  Director  General  Naval  Policy  and 
Warfare  (DGNPW)[Ref.  55]  within  the  DCNS  Division. 


Figure  13:  RAN  OT&E  Organisation  post  May  1992 


So,  as  a  result  of  reorganisations  over  the  past  five  years,  the  RAN  has  no 
T&E,  let  alone  OT&E,  policy  maker  and  overseer.  It  also  has  no  OT&E  coordinator 
within  Navy  Office  and  the  RAN’s  OT&E  authority  is  buried  within  the  in-service 
engineering  management  area  of  the  Maritime  Con.mand.  The  OT&E  authority, 
however,  remains  independent  from  the  developing  and  production  agencies. 
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E.  RAN  OT&E  POLICY 

RAN  OT&E  policy  is  embedded  in  the  more  general  T&E  policy. 

1.  Policy  Documentation 

The  following  documents  outline  the  RAN  T&E  policy: 

a.  Capital  Equipment  Procurement  Manual  ( CEPMAN 1 ) 

Australian  DoD  T&E  policy  has  its  basis  in  the  Capital  Equipment 

Procurement  Manual  (CEPMAN  l)[Ref.  38],  the  aim  of  which  is  to  assist  project 
managers  in  assessing  the  need  for  T&E  in  a  project  and  to  provide  guidelines  for 
its  conduct.  The  document  provides  broad  T&E  policy  and,  other  than  defining 
OT&E,  little  policy  is  provided  as  to  its  conduct. 

b.  DI(N)  LOG  82-1,  RAN  Test  and  Evaluation  Policy 

The  top  level  T&E  policy  document  for  the  RAN  is  the  Defence 
Instruction  (Navy)  on  RAN  Test  and  Evaluation  Policy  [Ref.  44].  Its  purpose  is  to 
issue  policy  for  the  conduct  of  T&E  in  capital  procurement  and  modernisation 
projects  in  the  RAN. 

c.  ABR1921 

ABR  1921  [Ref.  41],  contains  T&E  and  acceptance  policy  applicable  to 
ships  building,  undergoing  modernisation,  conversion  or  extended  refit  Although  it 
refers  to  OT&E  under  PAT&E  Part  2,  it  does  not  address  OT&E  specifically. 

2.  T&E  Decisions 

RAN  T&E  policy  [Ref.  44:p.  2]  states  that  the  decision  to  subject  an 
acquisition  to  T&E  is  initially  decided  by  the  Sponsor  and  detailed  in  the  Capability 
Proposal.  The  nature  and  extent  of  the  T&E  is  then  decided  by  the  Project  Director 
in  consultation  with  the  E)esign  Approval  Authority  and  RANTAU.  Any  conflict 
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over  the  requirement,  nature  and/or  extent  of  the  proposed  T&E  is  resolved  by 
DCNS  and  ACMAT-N  for  operational  and  design  matters  respectively.  However,  in 
practice,  the  nature  and  extent  of  T&E  is  broadly  decided  by  the  Sponsor,  and  the 
Project  Director  is  responsible  after  consultation  with  the  DAA  and  the  Trials 
Authority  [Ref.  45]. 

3.  Independence. 

The  independence  of  a  testing  authority  is  recognised  in  general  terms  in 
CEPMAN  1  [Ref.  38]  which  recognises  that  independent  verification  and  validation 
(IV&V)  is  an  important  means  of  providing  early  detection  of  problems,  and  that  it 
can  have  value  and  credibility.  However,  it  implies  that  “independence”  means 
independent  from  the  Defence  Department,  i.e.,  contractors.  With  respect  to  OT&E, 
CEPMAN  1  compromises  the  independence  of  an  OT&E  authority  by  stating  that 
the  Design  Approval  Authority  (DAA),  which  is  part  of  the  Developing  Agency,  is 
responsible  for  the  assessment  of  the  acceptability  of  the  product  design  as  a  basis 
for  Acceptance  into  Service.  It  provides  for  the  DAA  being  involved  in  offering 
support  to  trials  establishments  (as  required)  during  OT&E  in  order  to  provide  DAA 
advice  to  the  Project  Manager  on  the  suitability  of  the  product  for  Acceptance  into 
Service.[Ref.  38:p.  14-7]  This  appears  to  be  more  applicable  to  services  other  than 
the  RAN,  as  the  DAA  within  Navy  does  not  advise  the  PM  on  the  suitability  for 
AINS. 

The  RAN  T&E  Policy  DI(N)  [Ref.  44:p.  3]  is  more  clear  when  it  states 
OT&E  is  to  be  conducted  progressively  by  an  authority  independent  of  the 
development  and  production  agencies  (usually  RANTAU).  With  the  demise  of 
RANTAU  and  the  transfer  of  OT&E  responsibilities  to  CTE  on  the  staff  of  the 
Maritime  Commander,  independence  from  the  development  and  production 
agencies  is  maintained. 
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4.  Use  of  Contractors 

Although  the  Australian  definitions  of  OT-3  and  0T4  (Table  9)  includes  the 
requirement  that  the  system  should  be  demonstrated  “using  normal  Fleet 
Personnel”,  the  documentation  makes  no  policy  statement  regarding  contractor 
personnel  involvement  in  system  operation  or  maintenance,  during  OT&E.  It  has 
been  RAN  practice  (e.g.,  in  the  DDG  Modernisation  Project)  to  have  contractors 
onboard  resolving  technical  problems  during  the  SQT  period,  which  is  included  as 
part  of  OT-3  in  the  RAN. 

5.  T&E  Planning 

The  RAN  uses  the  Test  and  Evaluation  Master  Plan  (TEMP)  as  the  single, 
executive,  long  range  planning  document  for  T&E  in  a  project.  The  TEMP  is  to  be 
raised  in  draft  form  as  early  as  practicable.  Based  on  the  USN  format,  it  is  the 
ultimate  responsibility  of  the  project  sponsor  because  it  is  the  primary  tool  to  ensure 
the  user  requirement  is  met.  However,  during  the  period  when  there  is  a  full  time 
Project  Office,  TEMP  coordination  is  normally  carried  out  by  the  T&E  Manager  in 
the  Project  Office.  RANTAU  (now  CTE)  as  the  OT&E  authority  provides  the  OT&E 
input  The  TEMP  is  endorsed  by  all  affected  organisations  and  is  released  jointly  by 
the  Deputy  Chief  of  Naval  Staff  (DCNS),  and  the  Assistant  Chief  of  Naval  Staff- 
Materiel  (ACMAT-N),  as  the  Developing  Agency.  If  any  issues  remain  unresolved 
at  this  level,  the  TEMP  will  be  resolved  by  the  Chief  of  Naval  Staff  (CNS).  [Ref. 
44:pp.  4-5]  The  authority  for  DCNS  to  release  the  TEMP  was  based  on  his  role  as 
Project  Sponsor.  With  HQADF  now  assuming  this  role,  the  DCNS  function  in 
TEMP  development  is  unclear. 
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6.  Combined  DT&E  and  OT&E 


Although  not  sped  heal  iy  addressed  as  such,  combined  DT&E,  (PAT&E) 
and  OT&E  is  implied  as  being  condoned  and  almost  mandatory  within  the  RAN. 
The  T&E  DI(N)  [Ref.  44;p.  3]  states  that  OT-3  and  OT-4  will  incorporate  the 
requirements  of  PAT-6  and  PAT-7  Inspection  Tests  and  Trials  as  described  in  ABR 
1921.  It  also  states  that  OT-3  will  be  conducted  during  Ship  Qualification  Trials.  No 
limitations  or  guidance  as  to  the  possible  hazards  of  this  approach  are  addressed. 

7.  Modeling  &  Simulation 

The  use  of  models  and  simulation  (M&S)  in  OT&E  are  not  addressed  by 
the  RAN  documentation,  and  their  application  to  OT&E  has  been  limited. 
Simulation  and  stimulation  were  used  extensively  during  DT&E  by  both  the 
Submarine  Warfare  Systems  Centre  (SWSC)  and  the  Combat  Data  System  Centre 
(CDSC)  for  the  Submarine  Weapons  Update  Program  (SWUP)  and  DDG 
Modernisation  respectively.  However,  Major  combatant  OT&E  applications  of 
M&S  within  the  RAN  have  been  limited  to  the  DDG  Modernisation  Project.  M&S 
developed  by  the  Maritime  Systems  Division  of  the  Defence  Science  and 
Technology  Organisatic«i  were  used  by  RANTAU  for  the  evaluation  of  the  upgraded 
combat  and  weapons  systems  in  the  Modernised  DDGs. 

8.  Foreign  Weapons  Evaluation 

Despite  the  most  recent  Australian  Government  “White  Paper”  on  Defence 
Policy  [Ref.  6:p.  70]  requiring  the  need  to  be  able  to  determine  the  performance  in 
the  Australian  environment  of  equipment  of  both  overseas  and  local  origin,  no  DoD 
or  RAN  T&E  policy  specifically  addresses  the  evaluation  of  foreign  systems. 
Although,  almost  all  of  the  systems  procured  by  the  RAN  in  recent  years  are 


“foreign”  in  origin,  the  RAN  documentation  focuses  on  the  T&E  process  for  the 
progressive  evaluation  of  locally  developed  systems. 

9.  OT&E  Funding 

OTifeE  is  funded  separately  within  a  project  and  is  estimated  and  bid  for  by 
RANTAU  to  Project  Directors.  [Ref.  44:p.  6] 

10.  Land  Based  Test  Sites 

The  guidance  for  OT&E  using  land  based  test  sites  includes  a  statement  that 
system  centres  and  simulators  will  be  employed  for  early  stages  of  OT&E  if 
available  [Ref.  44:p.  4],  and  that  OT-2  may  be  conducted  at  a  land-based  test  site 
[Ref.  44:p.  3]. 

11.  Ship  OT&E 

The  RAN  recognises  that  some  major  acquisition  projects  will  be  treated 
differently  to  others  in  so  far  as  protracted  construction  time  usually  precludes  T&E 
of  a  prototype  before  the  decision  to  proceed  to  production.  The  degree  of  technical 
risk  will  be  assessed  in  each  case  and,  where  considered  sufficient,  the  decision 
taken  to  use  land-based  test  sites  and/or  prove  a  particular  system  in  another 
platform.  When  this  occurs,  DT-2  and  OT-2  shall  be  conducted  at  the  test  site  and 
the  “whole  ship”  evaluation  undertaken  as  soon  as  practicable  after  delivery. 

F.  RAN  SHIP  ACCEPTANCE 

1.  Policy 

The  decision  to  accept  a  new  or  modified  ship  into  the  RAN  is  vested  in  the 
Chief  of  Naval  Staff  (CNS).  An  Acceptance  Board  is  established  to  provide  an 
assessment  of  the  ship  and  submit  recommendations  upon  which  CNS  can  base  his 
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acceptance  decision.  The  assessment  of  a  ship  leading  to  an  acceptance  decision  has 
two  distinct  components  [Ref.  41  :p.  15.7]: 

•  Determining  where  the  ship  as  built  and  tested  does  not  meet  the  Agreed  Ship 
Characteristics,  contracted  performance  specifications  and  approved  support 
standards. 

•  Deciding  whether  those  Agreed  Ship  Characteristics,  performance 
specifications  and  support  standards  accord  with  current  policies  standards 
and  practices. 

The  Acceptance  Board  (AB)  is  an  independent  body  established  by  CNS  to 
make  recommendations  for  CNS’s  acceptance  into  RAN  service  of  new 
construction  or  modernised  ships,  submarines,  aircraft,  and,  where  directed,  new 
systems,  installations  and  equipment.  Usually  established  for  each  new  ship  type 
and  only  for  first  of  class,  a  Board  is  composed  of  a  President,  a  Vice  President  (VP), 
and  additional  members  selected  from  appropriate  specialist  areas.  The  President, 
VP  and  Board  members  serve  on  the  Board  part  time.  The  Board  is  administered  by 
the  Director,  RANTAU  (now  CTE)  who  is  an  ex  officio  member,  assisted  by  a  full 
time  Secretary. [Ref.  42:pp.  1-3]  The  AB’s  sole  function  is  to  advise  CNS  on 
acceptance  matters,  and  it  is  essentially  a  board  of  review. 

The  RAN  divides  acceptance  into  three  phases: 

•  Preliminary  Acceptance 

•  Acceptance  into  Naval  Service 

•  Operational  Acceptance 

2.  Definitions 

The  “Delivery  Commissioning  and  Acceptance”  chapter  of  Reference  41 
provides  the  definitions  for  the  acceptance  process: 

•  “Delivery”  occurs  when  the  ship  is  contractually  delivered  from  the 


contractor’s  ownership  to  the  Government. 

•  “Preliminary  Acceptance”  is  an  assessment  by  the  RAN  to  determine  if  the 
ship  has  reached  both  material  and  support  status  sufficient  to  safely  and 
effectively  proceed  with  the  Navy  trials  and  evaluation  program.  It  usually  is 
concurrent  with  Delivery. 

•  “Acceptance  into  Naval  Service”  (AINS)  signifies  the  acceptance  of  the  ship 
as  a  fully  operational  unit  of  the  Fleet,  based  on  operational  performance  and 
support  levels  obtained  at  that  time. 

•  “Operational  Acceptance”  is  carried  out  only  on  the  first  of  class  to 
supplement  the  AINS  assessment  using  additional  operational  and  support 
experience  gained  with  the  Fleet  to  refine  the  vessel  and  Naval 
Requirements. 

3.  Responsibilities 

Delivery  of  the  ship  to  the  Government  is  a  contractual  matter  and  is 
managed  by  the  Project  Director.  Once  delivered,  the  process  is  one  of  ‘offer  and 
accept’  whereby  the  ship  is  offered  by  the  Project  Director  to  CNS  for  acceptance. 
RANTAU’s  responsibility  is  to  conduct  the  Acceptance  Trials  and  report  the  results. 
The  Acceptance  Board  does  not  conduct  tests,  trials  or  inspections,  but  may  witness 
them  if  required.  [Ref.  42]  The  Acceptance  Board  roles  are  as  follows: 

•  Be  satisfied  that  a  comprehensive  and  properly  supervised  series  of 
inspections,  tests,  trials  and  evaluations  have  been  carried  out  to  the  point 
where  assessments  can  be  made  on  current  suitability  for  operational  service 
on  the  basis  of  the  approved  current  requirements. 

•  Assess  the  ship,  aircraft  or  equipment,  by  examining  reports  of  materiel 
inspections,  operational  trials  and  evaluations,  attending  trials  as  required 
and  calling  for  such  other  tests  or  evaluations  as  necessary. 

•  Be  satisfied  that  current,  relevant  and  approved  documents  when 
promulgated  describe  the  required  operational  characteristics  against  which 
the  item  for  acceptance  can  be  tested  and  evaluated  for  operational 
suitability,  operational  effectiveness  and  supportability  in  service. 


4.  Procedure 


The  completion  process  for  a  warship  entering  service  is  outlined  in  Figure 
14.  This  process  may  be  divided  into  two  distinct  parts: 

•  Those  events  leading  to  and  culminating  in  delivery  /  preliminary  acceptance. 

•  Subsequent  activities  leading  to  AINS  and  Operational  Acceptance  (if  first  of 
Class) 


Figure  14:  RAN  Ship  Acceptance  Schedule  and  Milestones 

a.  Delivery  /  Preliminary  Acceptance 

Delivery  of  the  ship  to  the  Government  is  a  contractual  matter  and  is  the 
prerogative  of  the  Project  Director  acting  on  behalf  of  ACMAT-N  and  the 
Commonwealth  <£  Australia  [Ref.  41  :p.  15.3].  Immediately  prior  to  delivery,  the  AB 
will  advise  CNS  whether  the  ship  has  reached  a  materiel  and  support  status  suitable 
for  Preliminary  Acceptance.  Preliminary  Acceptance  assures  CNS  that  the  vessel  is 
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considered  safe  for  a  Naval  crew  to  take  control  of  the  vessel  from  the  contractor  and 
conduct  outstanding  RAN  trials  of  equipment  at  sea  or  undertake  specific  trials  for 
AINS  assessment.  [Ref.  41  :p.  15.10] 

b.  Acceptance  into  Naval  Service 

Acceptance  into  Naval  Service  is  essentially  the  acceptance/transfer  of 
the  ship  from  the  acquisition  part  of  the  Navy  (ACMAT-N)  to  the  inservice 
operations  and  support  areas,  as  having  met  the  user  requirement.  The  AINS 
assessment  is  carried  out  on  completion  of  the  Post  Delivery  Availability  (PDA) 
Trials  Period,  at  which  point  the  ship  should  be  in  a  materiel  state  ready  for 
operational  service.  Post  delivery  trials  leading  to  AINS  include  the  SQT  period, 
with  harbour  and  sea  elements.  While  the  PD  is  responsible  for  the  PAT&E 
program,  RANTAU  is  responsible  for  the  detailed  planning  and  witnessing  of  the 
trials  program  consulting  with  the  PD.  During  these  trials,  the  PD  may  provide  a  full 
time  trials  coordinator  onboard  to  assist  the  Commanding  Officer  and  the  other  trials 
authorities.  [Ref.  41:pp.  15.10  -  15.12]  On  completion  of  the  trials  period,  the  AB 
will  advise  CNS  whether  the  materiel  and  support  state  of  the  ship  is  sufficient  for 
it  to  enter  service  as  an  operational  unit  of  the  Fleet,  and  hence  it  can  be  accepted 
into  naval  service  [Ref.  42:p.  4]. 

c.  Operational  Acceptance 

For  a  first  of  class  ship  it  will  normally  not  be  possible  to  trial  and 
evaluate  all  aspects  of  the  operational  performance  and  maintainability  prior  to 
AINS  without  the  duration  of  the  PAT&E  Part  2  program  becoming  excessive  and 
because  certain  aspects  can  only  be  evaluated  in  the  operational  environment.  1  his 
follow-on  T&E  period  is  conducted  following  AINS  and  is  compiled  by  RANTAU 
in  consultation  with  the  Naval  and  Maritime  Headquarters  Staff.  It  comprises  a 
series  of  trials  and  evaluations  to  assess  operational  effectiveness  and  operational 
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suitability.  [Ref.  41:p.  15.14]  Typical  examples  of  these  activities  in  the  FOT&E 
program  are; 


•  the  evaluation  of  the  ship  as  part  of  a  force  during  an  exercise,  in  order  to 
develop  standard  operating  procedures  and  tactics,  and  to  explore  the  limits 
of  capability,  etc. 

•  evaluation  of  the  maintainability  and  stores  support  for  equipments  and 
systems,  to  determine  whether  or  not  the  complement,  stores  allowances, 
documentation,  etc.  are  adequate. 

Typically,  the  Follow-on  T&E  program  will  last  from  12-24  months, 
depending  on  the  class  of  ship.  At  the  end  of  this  period  RANTAU  compiles  a  report 
in  consultation  with  Maritime  Headquarters  Staff,  the  ship  and  other  authorities  as 
appropriate,  indicating: 

•  The  results  of  the  trials  and  evaluations 

•  Whether  any  changes  are  recommended  to  the  proposals  by  the  PD  for 
correcting  deficiencies  which  were  extant  at  the  end  of  PAT&E  Part  2  trials. 

•  What  additional  items,  if  any  should  be  added  to  the  list  of  deficiencies  and 
what  action  is  proposed  to  overcome  them. 

The  AB  assesses  the  results  and  provides  the  overview  to  ensure  the 
program  is  comprehensive.  This  assessment  serves  to  confirm  or  modify  the 
assessment  at  AINS. 


G.  RAN  OT&E  PROCEDURES 

Procedures  are  the  methodology  by  which  policy  is  carried  out.  This  section 
discusses  the  procedures  adopted  by  the  RAN  in  implementing  OT&E  policy. 
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1.  User  Requirement  to  the  Acceptance  Management  Plan. 

The  development  of  the  user  requirement,  and  its  subsequent  progression 
through  to  the  OT&E  test  documentation  (Figure  15)  is  through  the  following 
documents: 


Operational  Concept  Paper  (OCP) 

Defence  Force  Capability  Proposal  (DFCP) 
Required  Operational  Characteristics  (ROC) 
Approved  Ship  Characteristics(ASC) 

Test  and  Evaluation  Master  Plan  (TEMP) 
Acceptance  Management  Plan  (AMP) 


FULLSOLE 

PROGRAM 

ENGINfERING 

WOOUCTION 

PRELIM  Air«  OA 

INITIAnON 

CCVE1X)PMENT 

APPROVAL 

APntOVAL 

ACCEPT 

PHASE  2 

PHASE  3 

PHASE  4 

PHASE  5 

DEMONSIRATION 

AND 

EVALUATION 

FULLSCALE 

ENGINEZXING 

DEVELOPMENT 

PRODUCTION 

IN-SERVICE 

Figure  15:  Progressive  Definition  of  OT&E  Documentation 


a.  Operational  Concept  Paper  (OCP) 

The  strategic  basis  of  papers  (DOA-87  [Ref.  6]  and  ASP-90  [Ref.  5]) 
provide  endorsed  Government  policy  guidance  for  force  structure  planning.  These 
documents  identify  ADF  roles  and  establish  broad  priorities  for  capability 
development.  From  this  guidance.  Operational  Concept  Papers  are  produced  by 
HQADF  to  identify  the  way  ahead  for  specific  force  structure  issues.  [Ref.  54:p.  A-4] 

b.  Defence  Force  Capability  Proposal  (DFCP) 

Against  the  background  of  an  OCP,  a  Defence  Force  Capability 
Proposal  (DFCP)  is  developed  by  HQADF.  The  DFCP  is  the  document  against 
which  specifications  are  produced  and  ultimately  the  proposed  solution  is  evaluated 
as  being  suitable  for  service  use.  [Ref.  54:p.  A-5] 

c.  Required  Operational  Characteristics  (ROC) 

To  meet  the  overall  requirements  of  the  DFCP,  more  specific 
requirements  may  be  described  in  the  Required  Operational  Characteristics. 

d.  Agreed  Ship  Characteristics  (ASC) 

More  detailed  requirements,  usually  for  ship  programs,  are  developed 
into  the  Agreed  Ship  Characteristics  (sometimes  termed  Approved  Ships 
Characteristics).  Other  subordinate  performance  documentation  may  be  developed 
also. 

e.  Test  and  Evaluation  Master  Plan  (TEMP) 

For  a  new  ship  design,  the  standards  against  which  the  ship  is  assessed 
are  those  derived  from  the  then  approved  DFCP,  ship  characteristics  and  policies. 
These  standards  are  developed  into  a  TEMP  which  details  the  responsibilities  for  the 
specification  of  requirements,  objectives,  criteria  and  conduct  of  T&E,  together  with 
the  schedule  of  activities,  resources  and  key  contractual  dates.  The  most  important 
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part  of  the  TEMP  is  the  statement  of  Critical  Operational  Issues  (COI)  which  are 
either  of  high  technical  risk  or  vital  to  the  continuing  development  process.  These 
issues  may  not  be  dependent  on  thresholds,  but  should  cover  all  areas  that  affect  the 
systems  capability  to  accomplish  its  mission  in  a  combat  related  environment.  [Ref. 
44:p.  5] 

f.  Acceptance  Management  Plan  (AMP) 

The  Trials  Authority  develops  management  plans  for  the  trials  it 
conducts.  Where  those  trials  lead  to  AINS,  the  management  plan  is  termed  the 
Acceptance  Management  Plan.  The  AMP  takes  the  COIs  from  the  TEMP  and  details 
the  tests  and  trials  necessary  to  evaluate  them  to  support  an  AINS  decision. 
Similarly  a  Follow  on  OT&E  (FOTE)  Management  Plan  may  be  produced  detailing 
the  trials  for  the  OT-4  phase.  The  detailed  trials  plans  are  developed  from  these 
management  plans. 

g.  Summary 

As  in  the  USN,  the  hierarchy  of  requirements  documentation  in  the 
RAN  provides  the  linkage  from  capability  requirement  to  operational  test  program 
necessary  to  evaluate  the  final  product  against  the  user  requirement. 

2.  Assessment  of  Operational  Effectiveness 

The  RAN  has  found  it  difhcult  to  define  exactly  what  is  required  to  assess 
operational  effectiveness.  The  RANTAU  guidance  on  the  conduct  of  OT&E  [Ref. 
53]  recognises  that  the  user  requirement  is  the  bench  mark  for  determining  the 
degree  to  which  a  product  is  effective.  Also  that  operational  effectiveness  is  best 
assessed  by  a  performance  demonstration  by  normal  operating  personnel  in  the 
normal  or  given  environment.  But  otherwise  provides  very  little  guidance  on  the 
assessment  of  operational  effectiveness.  Techniques  employed  involve  analysing 
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each  COI  and  employing  modified  Ship  Qualification  Trial  and  Fleet  Exercise 
Program  techniques  to  evaluate  them.  Use  has  also  been  made  of  simulation, 
weapon  analysis  and  exercise  analysis.  The  lack  of  suitable  range  facilities  has  also 
limited  the  assessment  of  operational  effectiveness. 

This  lack  of  operational  effectiveness  assessment  capability  in  the  RAN  is 
due  to  most  systems  being  acquired  from  large  parent  Navies.  Consequently,  the 
RAN  has  seldom  had  the  need  to  assess  operational  effectiveness  for  itself.  The 
Submarine  Project  is  acquiring  a  variety  of  ranges  for  DT&E  applications,  e.g.,  the 
measurement  of  signatures,  manoeuvreing  characteristics,  weapon  control 
characteristics,  etc.  These  ranges  may  also  be  suitable  to  support  OT&E. 

3.  Assessment  of  Operational  Suitability 

The  assessment  of  operational  suitability  in  the  RAN  has  been 
controversial,  particularly  the  assessment  of  supportability.  This  quote  from  the  then 
Director  of  Naval  Integrated  Logistic  Support  Management  [Ref.  40]  in  1987 
highlights  a  particular  viewpoint: 

While  there  are  clearly  visible  yardsticks  by  which  production  or  weapon 
system  performance  can  be  measured,  the  success  of  logistic  planning  is  to  an 
extent  invisible.  Shortcomings  in  the  assessment  and  provision  of  support  wilt 
only  become  evident  due  to  operational  failure.  Alt  that  RANTAU  or  the  Ship 
Acceptance  Board  can  do  is  make  a  subjective  judgement  as  to  whether  all 
necessary  ILS  elements  have  been  addressed  and  ensure  that  an  acceptable 
follow  on  logistic  support  and  enhancement  process  has  been  set  in  place. 

Before  1987,  the  assessment  of  ILS,  a  subset  of  operational  suitability,  was 
rather  subjective.  The  assessment  of  supportability,  for  example,  consisted  of  a 
review  of  allowance  lists  by  experienced  technical  specialists  where  potential 
deficiencies  were  highlighted.  The  ILS  community  were  concerned  that  the  onboard 
spares  allowances  developed  by  detailed  logistic  support  analysis,  were  being 
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subject  to  a  relatively  simplistic,  qualitative  assessment.  Similariy  the  assessment  of 
Reliability  Maintainability  and  Availability  (RM&A)  was  superficial  with  only 
those  defects  noted  during  the  operational  a'^sessment  period  (from  five  to  seven 
weeks)  being  analysed.  A  new  approach  was  taken  in  1987  to  the  assessment  of 
operational  suitability,  and  to  ILS  in  particular.  The  assessment  process  has 
continued  to  be  refined  with  the  areas  in  Table  1 1  now  assessed: 


TABLE  11:  SUITABILITY  ISSUES  FOR  ASSESSMENT 
Suitability  Issues 

Availability 

Reliability 

Maintainability 

Safety 

Human  Factors 
Interoperability 
Compatibility 

Integrated  Logistic  Support  including: 

Maintenance  Planning 

Supply  Support 

Support  and  Test  Equipment 

Technical  Data  and  D<xumentation 

Manpower  and  Personnel 

Training  and  Training  Devices 

Facilities 

Packaging,  Handling,  Storage  and  Transport 
Computer  Support 
Configuration  Control 


a.  Suitability  Assessment  Procedure 

Safety  is  assessed  by  inspection  and  observation  during  system 
operation.  Human  factors,  interoperability  and  compatibility  are  assessed  by 
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observation  during  operational  effectiveness  testing.  The  assessment  of  ILS  is 
conducted  in  two  phases: 

( 1 )  ILS  Structural  Review.  Conducted  prior  to  operational  testing,  this 
review  assesses  if  the  ILS  intended  to  provided  will  support  the  ILS  policy  of  the 
Project.  It  consists  primarily  of  a  review  of  the  ILS  Plan  and  discussions  with  the 
functional  areas  handling  the  support. 

(2)  ILS  Operational  Review.  During  the  operational  testing  period,  the 
ship’s  ILS  related  documentation  and  records  are  reviewed,  maintainability 
demonstrations  are  performed  and  a  supportability  assessment  is  conducted.  The 
supportability  assessment  still  consists  of  a  review  of  the  allowance  lists  by 
experienced  technicians,  but  now  it  also  includes  an  analysis  of  spares  usage  and 
availability,  from  the  time  records  are  commenced  to  the  end  of  the  operational 
testing. 

RM&A  is  assessed  by  the  collection  and  analysis  of  data  during  the  OT- 
3  phase  of  operational  testing  to  give  an  indication  of  performance  prior  to  AINS, 
and  then  continued  for  12  months  during  the  OT-4  phase.  The  RAN  is  currently 
developing  a  routine  in-service  RM&A  data  collection  and  analysis  system, 
however,  until  that  is  operational,  the  data  is  collected  and  analysed  on  an  ‘as 
required’  basis  by  the  Trials  Authority. 

4.  OT&E  Reporting 

RANTAU  coordinates  the  issue  of  trials  reports  which  are  issued  on 
completion  of  each  segment  of  OT&E  e.g.,  o/c  ASW  phase  of  OT-3  etc.  Afinal  trials 
report  is  issued  to  the  AB  for  inclusion  in  their  AINS  report. 
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5.  OT&E  Coordination 

A  T&E  Planning  Group  is  formed  prior  to  TEMP  development  under  the 
leadership  of  the  T&E  manager  for  the  project.  Foundation  members  of  the  TEPG 
are  the  Project  Director,  Project  Sponsor,  the  Design  Approval  Authority  and 
RANTAU,  with  other  parties  seconded/called  upon  as  required.  [Ref.  44:p.  5]  The 
TEPG  analyses  T&E  requirements  and  estimates  resources  required  to  meet  the 
requirements. 

6.  OT&E  Personnel  Selection  and  Training 

Although  OT&E  is  a  specialised  discipline  with  its  own  philosophy  and 
methodology,  the  RAN  has  no  training  courses  on  OT&E,  or  on  T&E  in  general.  A 
number  of  officers  have  completed  the  USN  Operational  Test  Directors  Course  in 
recent  years  which  has  improved  the  knowledge  and  understanding  of  OT&E  within 
CTE’s  organisation  considerably.  These  courses  have  been  arranged  on  an  ad-hoc 
basis  through  overseas  visit  submissions  rather  than  as  pre-requisite  courses  for 
particular  billets.  No  special  selection  criteria  is  involved  for  posting  OT&E 
personnel  or  project  T&E  managers.  Although  some  personnel  may  develop  into 
OT&E  specialists,  no  career  path  is  consciously  provided. 

H.  RAN  OT&E  IMPLEMENTATION 

The  implementation  of  OT&E  policy  within  the  RAN  maybe  demonstrated  by 
reviewing  examples  of  ship  OT&E  conducted  to  date,  and  that  planned  for  ships  due 
to  enter  service  in  the  near  future. 

1.  DDG  Modernisation  Project 

The  RAN  has  three  DDGs  built  in  the  USA  to  a  modified  “Charles  F. 
Adams”  (DDG-2  class)  design.  These  ships  underwent  a  major  modernisation  and 
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refit  over  the  period  1987-1991.  The  major  modernisation  occurred  with  the  ships’ 
combat  system,  command  and  control  system,  sensors  and  weapons,  with  the  aim  of 
increasing  the  ‘supportability’  of  the  systems  rather  than  increasing  the  ships’ 
capability  [Ref.  39].  The  upgrade  was  similar  to  that  undertaken  on  a  small  number 
of  USN  DDGs.  Most  of  the  systems  were  imported  and  installed  locally,  however, 
the  combat  system  operational  program  was  developed  in  Australia.  The  risk  of  the 
project  failing  to  meet  operational  requirements  was  low,  but  the  RAN  needed  to 
quantify  the  operational  effectiveness  and  operational  suitability  of  the  modernised 
DDG. 

OT&E  consisted  of  an  OT-3  assessment  during  a  normal  SQT  period,  prior 
to  workup  and  the  Operational  Readiness  Evaluation.  This  was  followed  by  an  OT- 
4  phase,  termed  a  Combat  System  Evaluation  (CSE),  for  the  first  of  class.  This  OT- 
4  was  event  based  and  was  completed  over  a  12  month  period,  integrated  with  the 
normal  ships  program.  Procedures  developed  during  the  OT&E  for  the  DDG 
Modernisation  included  the  use  of  the  Combat  System  Trainer  (CST)  van  for 
simulating  complex  AAW  engagements,  the  use  of  system  modelling  and  analysis 
by  the  Weapons  Systems  Research  Laborat<xy  (WSRL),  and  the  use  of  new  ILS  and 
RM&A  data  collection  and  assessment  procedures. 

DDG  Modernisation  was  a  low  operational  risk  project.  The  OT&E 
followed  the  established  RAN  policy,  however,  it  was  the  first  to  actually  implement 
an  OT-4  period  to  more  fully  assess  operational  effectiveness  and  operational 
suitability  leading  to  OA. 

2.  Australian  Frigate  Project 

Two  US  FFG-7  class  frigates,  known  as  Australian  Frigates  (AP^,  are  being 
built  in  Australia.  The  frigates,  FFG-06  and  FFG-07,  are  being  built  to  the  HMAS 
DARWIN  FFG-04  (USN  FFG-44)  configuration  baseline  except  that  the  AN/SQS- 
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56  sonar  is  replaced  with  the  Australian  Mulloka  sonar.  Changes  to  the  configuration 
were  kept  to  a  minimum  commiserate  with  safety  and  operational  improvements 
approved  for  incorporation  into  FFG-04.[Ref.  51:p.  ix]  Ail  the  identified  equipment/ 
system  differences  between  HMAS  DARWIN  and  the  AFs  had  undergone  T&E 
prior  to  installation  in  FFG-05. 

As  the  AFs  have  an  almost  identical  configuration  to  the  last  FFG-7  class 
purchased  from  the  U.S.,  the  major  risk  was  in  the  production,  not  in  the  operational 
effectiveness  and  suitability  of  the  finished  product,  hence  OT&E  was  limited  to  an 
OT-3  period.  The  objective  of  OT&E  for  the  AFs  is  to  ensure  that  the  user 
requirements  have  been  met,  and  to  provide  an  estimate  of  operational  effectiveness 
and  suitability  in  the  RAN  operating  environment  prior  to  release  of  the  ship  to 
operational  service.  The  OT&E  comprises  an  OT-3  period,  which  assesses  the  AF 
against  the  baseline  established  by  HMAS  DARWIN,  including  any  approved 
changes.  [Ref.  51  :p.  10]  The  scope  of  OT-3  includes  a  Light  Off  Examination,  Test 
Procedures,  CSSQT  and  Post  PSA  Trials.  OT-3  is  being  conducted  on  each  ship  and 
involves: 

•  Installation  Inspections 

•  Harbour  Phase  System  Qualification  Trials 

•  Sea  Phase  System  Qualification  Trials 

•  Communications  System  Operability  Trials 

•  Integrated  Logistic  Support  Assessments 

•  Fleet  Inspections 

•  Air  Certification 

•  Light  Off  Examination 
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As  the  significant  operational  difference  in  the  AFs  is  the  Mulloka  sonar,  an 
Operational  Performance  Demonstration  (OPD)  was  planned  to  determine  if  the  AF 
can  fulfil  it’s  ASW  roles  and  functions  with  the  Mulloka  sonar.  One  week  of  trials 
was  planned  during  the  OT-3  period  to  ensure  that  the  Mulloka  sonar  system  had 
been  correctly  installed  and  met  its  operational  performance  characteristics.  This 
period  was  found  to  be  insufficient,  primarily  due  to  technical  problems,  and  a 
“mini-OPEVAL”  is  now  planned  to  quantify  the  system’s  performance. [Ref.  61;p. 
A-2]  The  acceptance  schedule  (Figure  16)  follows  established  RAN  policy. 
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Figure  16:  Australian  Frigate  Acceptance  Schedule 


The  TEMP,  governing  all  tests,  trials,  and  evaluations  up  to  AINS,  was 
produced  as  an  Australian  Frigate  Addendum  to  the  U.S.  FFG-7  Class  TEMP.  A  Post 
Delivery  Test  and  Trials  Plan  [Ref.  S2]  was  produced  to  serve  as  an  advance 
planning  document  and  an  implementation  plan. 


An  Acceptance  Board  was  established,  initially  for  FFG-05  only.  The 
PDT&T  Plan  includes  the  requirement  that  the  AB  is  responsible  for  providing  the 
Final  Contract  Trials  agenda  and  conducting  the  trials[Ref.  52:p.  3-17],  which  is  not 
in  accordance  with  RAN  AB  policy.  In  its  report  at  AIN  S,  the  AB  considered  that  the 
application  of  the  AB  process  was  worthwhile  and  recommended  that  it  should  be 
followed  for  FFG-06,  which  is  contrary  to  RAN  policy  of  establishing  the  Board  for 
first  of  class  only.  The  Board  also  noted  that  acceptance  documentation  was 
contradictory  and  unclear.  [Ref.  61:p.  2] 

The  AF  is  a  low  operational  risk  project,  except  for  the  addition  of  the 
Mulloka  sonar.  The  OT&E  followed  the  established  RAN  policy,  however,  it  failed 
to  plan  an  OT-4  period  to  more  fully  assess  operational  effectiveness  and  operational 
suitability  of  the  Mulloka  sonar.  An  OPEVAL  to  fully  assess  the  performance  of  the 
sonar  has  now  been  included. 

3.  ANZAC  Ship  Project 

Australia  and  New  Zealand  are  cooperating  in  the  building  of  the  ANZAC 
frigate.  Essentially  a  Blohm  and  Voss  MEKO  200  design  incorporating  a 
combination  of  European  and  US  equipment,  the  design  is  classified  as  low  risk, 
although  modifications  to  the  design  were  necessary  to  meet  Australian 
requirements  and  to  facilitate  construction  in  Australia.  [Ref.  58] 

a.  ANZAC  Ship  OT&E 

OT&E  for  the  ANZAC  ships  is  currently  planned  to  include: 

(1)  OT-2  during  which  RANTAU  may  witness  specific  contractor 
testing,  and  elect  to  conduct  initial  operational  evaluation  using  the  facilities 
provided  as  part  of  the  Combat  System  Support  Centre,  a  LBTS  established  initially 
for  the  conduct  of  DT&E,  and  then  to  provide  in-service  system  support. 


TABLE  12:  COLLINS  CLASS  POST-DELIVERY  TRIALS  [Ref.  59] 


ITT  Category 
(per  Contract) 

RANT&E 

Phases 

Subject  of  ITT /T&E 

CAT  6 

DT-3 

First  of  Class  Trials  -  Submarine  01 

OT-3 

OPEVAL  -  Submarine  01 

PAT-6 

Submarines  02  to  06 

CAT? 

CfTA 

Submarines  01,02  and  03  (possibly) 

a.  CAT  6  Testing 

CAT  6  testing  consists  of  Safety  and  Operational  Work-ups,  First  of 
Class  Trials,  Operational  Certification  Trials,  and  OPEVAL. 

(1)  First  of  Class  Trials  (DT-3).  These  trials  will  establish  the  actual 
performance  of  the  COLLINS  design  and  the  in-service  safe  operating  limits  and 
conditions  for  the  class. 

(2)  PAT-6.  These  trials  will  be  the  Operational  Certification  Trials 
ccMiducted  after  delivery  in  support  of  AINS  cf  the  particular  submarine.  The  criteria 
will  be  based  on  applicable  thresholds  in  the  Agreed  Ship  Characteristics  and  the 
results  of  First  of  Class  Trials  with  submarine  01.  The  principal  events  in  PAT  6 
include: 

•  Safety  and  operational  work-up 

•  Manoeuvreing  trials 

•  Weapon  handling  and  discharge 

•  Combat  system  operation,  including  weapon  firings 

•  Signature  measurements 
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(3)  or -3.  These  trials  are  planned  to  evaluate  the  first  of  class  only, 
after  delivery,  conducted  in  order  that,  in  conjunction  with  the  results  of  PAT&E 
Part  2,  a  recommendation  for  AINS  may  be  made.  OT-3  includes  an  ILS  and  RM&A 
assessment,  and  an  OPEVAL  of  four  to  six  weeks  during  which  the  submarine  will 
be  tasked  to  conduct  a  series  of  operations  representative  of  the  mission  profiles 
against  simulated  threats  and  targets  of  interest.  These  trials  may  be  reduced  after 
experience  with  the  first  submarine. 

The  proposed  CAT  6  schedule  is  detailed  in  Figure  17. 


Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec  Jan 


Figure  17:  HMAS  COLLINS  -  Proposed  CAT  6  Schedule  [Ref.  49] 
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Of  note  is  that  the  OPEVAL  is  planned  to  be  conducted  following  the 
completion  of  DT-3,  work  up  and  the  Operational  Readiness  Evaluation  (ORE).  So 
it  will  be  conducted  on  a  materially  proven  and  operationally  worked  up  submarine. 

b.  CAT  7  Testing 

CAT  7  testing,  comprising  OT-4  exclusively,  will  consist  of: 

(1)  Long  Term  RM&A  and  ILS  Evaluation.  This  will  determine  with 
greater  statistical  c  onfidence,  the  Reliability,  Maintainability  and  Availability  of  the 
submarine  class,  and  assess  the  implementation  of  ILS  items. 

(2)  Measures  of  Effectiveness  (MQEs)  Testing.  Some  MOEs  will 
require  determination  by  statistically  significant  numbers  of  tests.  However,  to 
minimise  resource  requirements  the  tests  will  be  constructed  to  allow  the  total  test 
time  and  asset  requirements  to  be  minimised.  To  accommodate  other  scheduling 
limitations,  this  phase  will  probably  occur  in  a  number  of  stages. 

(3)  Major  Exercise.  This  phase  assesses  the  performance  of  the 
submarine  in  the  roles  required  of  it  during  a  major  exercise. 

(4)  Operational  Eteplovment.  This  phase  evaluates  the  performance  of 
the  submarine  and  the  support  infrastructure  during  an  operational  deployment  and 
maintenance  period  in  a  foreign  port. 

The  proposed  CAT  7  schedule  is  detailed  in  Figure  18. 
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Figure  18:  HMAS  COLLINS  >  Proposed  CAT  7  Schedule  [Ref.  49] 


c.  Use  of  Land  Based  Test  Sites 

The  submarine  project  includes  the  construction  of  a  combat  system 
simulator  and  propulsion  system  simulator,  both  for  system  development  and 
training  applications.  Some  OT&E  activities  may  be  conducted  by  GTE  on  these 
systems  at  the  land  based  test  sties  on  a  non-interfering  basis  to  the  contractor.  [Ref. 
59;p.  20] 

d,  Collins  Class  Suhnuuine  OT&E  Summary 

Funding  provision  for  RAN  T&E  up  until  AINS  has  not  been  included 
in  the  overall  financial  budgeting  for  the  New  Submarine  Project  [Ref.  59:p.  ii].  To 
assist  in  the  progressive  development  of  the  TEMP  and  to  plan  and  agree  those 
activities  that  will  be  required  for  AINS  and  OA,  a  TEPG  has  been  established.  The 
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detailed  planning,  preparation,  execution  and  reporting  of  RAN  T&E  activities  is  the 
responsibility  of  a  subordinate  T&E  Working  Group. 

The  Collins  class  submarine  possibly  has  the  highest  operational  risk  of 
all  current  major  RAN  combatant  projects.  Consequently  the  increased  need  for 
OT&E  has  been  recognised  by  CTE.  The  planned  OT&E  program  demonstrates  a 
departure  from  the  usual  RAN  OT&E  policy,  and  reflects  a  growing  awareness 
within  CTE  of  the  need  to  separate  DT  and  OT  activities,  and  to  include  a  dedicated 
OT-3  period  after  the  usual  SQT  and  ORE. 

5.  OT&E  Implementation  Summary 

These  examples  of  RAN  implementation  of  ship  OT&E  policy  illustrates  a 
number  of  key  issues: 

•  The  operational  risk  of  ship  projects  within  the  RAN  is  increasing  as  the  new 
RAN  combatants  have  a  unique  weapons  systems  configuration. 

•  OT&E  is  not  generally  conducted  to  support  the  production  or  purchase 
decision  of  ships. 

•  The  RAN  relies  on  the  results  of  a  combined  SQT  and  OT-3  period  to  provide 
a  basis  for  AINS. 

•  ILS  and  RM&A  assessments  are  becoming  more  effective  and  credible. 

•  Modeling  and  simulation  is  increasingly  being  used  to  assist  OT&E. 

•  Funding  of  ship  OT&E  is  not  being  included  in  original  project  estimates. 

•  Acceptance  and  T&E  documentation  is  contradictory  and  unclear. 

L  RAN  OT&E  ISSUES 
1.  What  is  OT&E? 

Although  the  term  “OT&E”  in  the  RAN  is  defined,  and  the  hierarchy  of 
Defence  and  RAN  documentation  addresses  OT&E,  the  policy  for  the  conduct  of 
OT&E  in  the  RAN  is  not  specific  and  is  included  with  general  T&E  policy. 


Combined  with  a  lack  of  distinction  between  DT-3,  PAT-6  and  C)T-3.  this  leads  to  the 
few  specific  OT&E  policy  statements  often  being  contradictory  and  unclear. 

a.  Independence  of  the  OT&E  Authority 

OT&E  is  specified  in  RAN  T&E  policy  [Ref.  44:p.  3]  as  being 
conducted  by  an  authority  independent  of  the  developing  and  production  agencies, 
and  the  results  of  OT-3  support  an  AINS  decision.  However,  CPO  Manual  [Ref. 
38:p.  14-7]  states  that  the  Design  Approval  Authority  (part  of  the  developing 
agency)  offers  support  to  trials  establishments  during  OT&E  in  order  to  provide 
advice  to  the  Project  Director  on  the  suitability  of  the  product  for  Acceptance  into 
Service.  This  leads  to  conflicts  between  the  DAA  and  the  Trials  Authority  during 
the  conduct  of  OT&E. 

b.  Confusion  between  DT‘3,  PATS  and  OT-3 

RAN  T&E  policy  [Ref.  44;pp.  2-3]  explains  the  T&E  phases  in  the 
procurement  prxx^ess  and  then  includes  a  note  stating  that  OT-3  and  OT-4  will 
incorporate  the  requirements  of  PAT-6  and  PAT-7  Inspection  Tests  and  Trials  as 
described  in  ABR  192 1  (Reference  41).  ABR  1921  however,  defines  many  tests  as 
being  PAT&E,  that  RANTAU  [Ref.  43]  define  as  OT&E.  Also  DT-3  is  shown  as 
being  conducted  during  the  same  period,  however,  the  philosophy,  aims,  objectives 
and  methodology  of  DT&E,  PAT&E  and  OT&E  are  totally  different.  Perhaps  this 
overlap  or  “blurring"  of  definitions  arose  because  RANTAU  was  not  only  the  RAN 
OT&E  authority,  but  also  the  RAN  T&E  authority  who  conducted  aspects  of  DT&E 
and  PAT&E  in  addition  to  OT&E.  With  a  large  part  of  its  workload  devoted  to 
conducting  trials  following  the  routine  refit  of  combatants,  RANTAU  conducted 
HATS  and  SATS  which  were  understood  to  be  PAT&E  and  then  went  on  to  conduct 
the  more  operational  Ship  Qualification  Trial  (SQT)  which  was  perceived  to  be 
OT&E.  The  SQT  period,  however,  is  rarely  a  free  play  environment,  makes  limited 
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use  of  countermeasures,  and  is  conducted  prior  to  the  crew  being  fully  worked  up. 
Also  contractors  are  often  still  onboard  making  final  adjustments  to  systems  and 
assisting  the  crew  with  operation  and/or  maintenance.  The  SQT  is  essentially  a 
PAT&E  function  which  may  include  some  components  of  the  operational 
effectiveness  and  suitability  aspects  of  OT&E. 

The  problem  with  this  “blurring”  of  definitions  goes  beyond  the 
semantics.  The  results  of  OT-3  are  used  to  support  an  AINS  decision,  but  as  it  is 
really  PAT&E,  it  assesses  more  the  ability  of  the  ship  to  meet  technical  or  Fleet 
standards.  It  does  not  truly  assess  the  operational  effectiveness  and  operational 
suitability  of  a  new  ship  against  the  user  requirement.  Therefore,  a  ship  is  not  truly 
subject  to  OT&E  against  the  user  requirement  before  it  is  granted  AINS. 

This  lack  of  distinction  between  T&E  types  also  confuses  Project 
Directors  and  other  Defence  managers.  It  leads  to  a  lack  of  consideration  of  the 
importance  of  OT&E,  and  hence  the  need  to  consider  OT&E  funding  early  in  a 
Project.  It  also  leads  to  confusion  over  T&E  management  responsibilities.  It 
confuses  the  role  of  the  Acceptance  Board,  which  is  really  assuming  a  quality 
control  function  over  what  is  essentially  PAT&E  testing,  rather  than  it’s  members 
using  their  knowledge  and  experience  in  assessing  the  results  of  true  OT&E  against 
the  user  requirement. 

The  difference  in  terminology  used  by  different  documents  and 
authorities  also  creates  confusion.  For  example,  the  term  OPEVAL  is  defined  [Ref. 
38;p.  14A-2],  but  terms  such  as  mini-OPEVAL  [Ref.  61],  Operational  Performance 
Demonstration  [Ref.  51]  and  (Ship)  Operational  Trials  [Ref.  41]  [Ref.  44]  are  not. 

2.  OT&E  and  Acceptance  of  New  Combatants 

With  the  construction  of  new  frigates  and  submarines,  the  RAN  has 
embarked  on  its  greatest  shipbuilding  program  since  the  Second  World  War. 
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Although  these  combatants  are  being  built  to  proven  designs  and,  therefore,  are 
considered  relatively  low  risk,  they  do  present  a  greater  operational  risk  than  recent 
RAN  combatant  programs.  In  September  1991,  CNS  expressed  concern  that  the 
RAN  “may  not  be  adequately  placed”  to  properly  manage  the  comprehensive  and 
complex  procedures  necessary  to  accept  the  ANZAC  ships  and  Collins  class 
submarines  into  service  [Ref.  62].  This  has  focussed  attention  on  the  acceptance 
procedures  and  consequently  on  the  OT&E  of  the  new  combatants. 

3.  The  New  RAN  OT&E  Organisation 

RANTAU  ceased  to  exist  as  an  organisation  on  May  15, 1992.  The  rationale 
leading  to  this  decision  is  important  as  it  is  indicative  of  the  level  of  understanding 
of  OT&E  within  the  RAN. 

a.  Background 

RANTAU  was  established  in  1966,  and  originally  administered  by 
Navy,  but  later  by  the  Defence  Scientific  and  Technical  organisation.  In  1982 
control  of  RANTAU  was  returned  to  Navy  with  the  stated  purpose  of  conducting 
operational  testing  and  evaluation,  conducting  inspections  and  providing 
operational  assessment  data.[Ref.  46;p.  1]  Functional  and  operational  responsibility 
for  RANTAU  was  transferred  from  the  DCNS  Division  of  Navy  Office  to  the 
Maritime  Commander  on  31  July  1991  concurrent  with  the  introduction  of  the 
Program  Management  and  Budgeting  System(PMBS).  The  Director  of  RANTAU 
was  made  responsible  to  the  Maritime  Commander  for  the  management  of 
RANTAU. 

b.  What  Did  RANT  A  U  Achieve? 

As  a  result  of  this  change  in  the  organisational  structure  (a  change  due 
fundamentally  to  financial  funding  and  accounting  requirements)  there  was  a  need 
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to  question  what  RANTAU  achieved  overall,  and  whether  its  tasking  should  be 
amended  to  more  accurately  reflect  the  needs  of  the  Maritime  Commander  in 
maintaining,  and  where  necessary  improving,  the  operational  capability  of  the 
Maritime  Force.  A  study  was  directed  [Ref.  46]  to: 

•  focus  on  the  tasks  required  to  be  undertaken  to  assess  the  operational 
effectiveness  of  Maritime  Force  units  and  the  Force  as  a  whole. 

•  study  any  aspects  of  RANTAU’s  current  employment  which  were 
inappropriate  given  its  functional  and  operational  responsibilities  to  the 
Maritime  Command. 

The  results  of  this  study  [Ref.  47]  found  that  only  about  20%  of  the 
work  carried  out  by  the  RANTAU  Trials  Unit  was  carried  out  for  the  Maritime 
Commander.  This  consisted  of  SQTs,  and  HATS  and  SATS.  The  large  majority 
(80%)  of  the  work  done  by  the  Trials  Unit  was  found  to  be  done  on  behalf  of  the 
Assistant  Chief  of  Naval  Staff  -  Materiel  (ACMAT-N)  in  accordance  with  ABR 
192 [(Reference  41).  The  report  stated  that  the  work  conducted  for  ACMAT-N 
included; 

•  assisting  with  the  writing  of  the  TEMP  for  every  project. 

•  installation  inspections  as  part  of  PAT &E  Part  1 . 

•  heavy  involvement  with  PAT&E  Part  2. 

•  the  conduct  of  OT&E  for  most  projects. 

•  support  for  the  post  delivery  acceptance  into  Naval  Service  (AINS) 
process.[Ref.  47:p.  17]. 

c.  Organisation  Options 

The  report  proposed  transferring  the  Trials  Unit,  less  a  proportion  to 
support  Fleet  Trials,  to  the  Materiel  Division.  The  unit  would  then  be  ideally  situated 
to  form  the  nucleus  of  an  organisation  with  responsibility  for  the  coordination  and 
administration  of  all  trials  conducted  for  and/or  by  the  RAN.  It  could  also  form  the 
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core  of  the  AB,  with  perhaps  the  Director  being  the  standing  vice  president  of  the 
body. 

The  Materiel  Division  rebutted  this  argument  with  the  perception  that 
an  independent  audit  authority  is  necessary  to  maintain  production  standards.  It 
appears  that  there  is  a  belief  that  if  the  RAM’s  trials  authority  was  to  be  a  functional 
unit  of  the  Materiel  Division,  then  the  head  of  the  Division  would  be  able  to 
influence  trials  results.  The  report  contends  that  this  argument  appears  to  point  to  a 
management  problem  rather  than  a  structural  issue,  but  in  any  case  it  did  nothing  to 
counter  the  argument  that  the  RANTAU  Trials  Unit  is  not  an  appropriate  component 
of  the  Operations  Sub-program  of  the  PMBS  (i.e.,  the  Maritime  Command).  A 
solution  to  this  problem  would  be  to  return  the  Trials  Unit  to  the  Executive  Sub¬ 
program  (i.e..  Navy  Office).  However,  it  recognised  that  the  service  provided  by  the 
trials  authority  was  also  largely  unrelated  to  that  sub-program.  The  report  then 
considered  the  impact  of  HQADF,  who  occasionally  tasks  RANTAU  in  support  of 
projects  for  the  development  of  the  maritime  component  of  the  ADF. 

The  report  recognised  that  this  seemingly  simple  organisation  of 
RANTAU  was,  in  fact,  complex  due  to  the  variety  of  tasks  imposed  upon  the  unit 
by  a  number  of  masters  on  a  day  to  day  basis.  It  concluded  that  as  a  single  unit, 
RANTAU  does  not  rest  comfortably  with  the  current  arrangement  and  nor  does  it 
with  any  other  Navy  Sub-program  for  budgeting  and  management  purposes. 

d.  An** Audit** Function 

The  report  states  that  RANTAU  served  an  apparent  independent  audit 
function  for  the  Materiel  Command,  and  that: 

this  function  was  something  that  had  evolved  over  time  rather  than  in  response 
to  any  specific  need.  It  was  beyond  the  scope  of  the  original  purpose  for  the 
organisation  which  became  RANTAU,  and  the  absolute  need  for  such  an  audit 
service  in  the  era  of  PMBS  and  tight  fiscal  control  is  questioned.  An  audit 
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service  for  the  Materiel  Division  does  not  seem  an  appropriate  sub-component 
for  the  Combat  Forces  sub-program. 

The  report  concluded  that,  other  than  for  a  small  trials  unit  to  conduct 
trials  in  support  of  units  assigned  to  the  fleet,  it  is  inappropriate  for  the  Maritime 
Commander  to  have  functional  and  administrative  control  of  a  Trials  Unit  which 
largely  serves  the  Materiel  Division.  The  options  appeared  to  be  the  transfer  of  the 
Trials  Unit  to  the  either  the  DCNS  or  Materiel  Divisions. 

e.  The  OT&E  Function 

The  report,  however,  failed  to  recognise  that  RANTAU  was  the  OT&E 
authority.  As  such,  the  “audit”  work  apparently  done  for  the  Materiel  Command  was 
really  done  on  behalf  of  the  sponsor  (i.e.,  HQADF)  and  the  user  (i.e.,  the  Maritime 
Command),  as  OT&E  is  the  final  feedback  that  the  item  produced  by  the  Materiel 
Command  meets  the  user  requirement. 

f.  Current  Trials  Organisation 

Following  the  release  of  this  report,  and  due  to  the  RAN’s  decision  not 
to  adequately  staff  and  fund  RANTAU  as  an  independent  unit,  the  functional  tasks 
of  RANTAU  were  incorporated  within  the  framework  of  the  Maritime  Headquarters 
[Ref.  48].  The  current  placement  of  the  former  Trials  Unit,  now  known  as 
Commander  Test  and  Evaluation,  within  the  Maritime  Command  is  shown  in  Figure 
19.  Of  note  is  that  the  OT&E  authority  is  now  responsible  to  the  Maritime 
Commander  through  the  in-service  engineering  management  chain  of  command. 
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Figure  19:  Commander  Test  and  Evaluation  Organisation 


J.  SUMMARY 

1.  Importance  of  OT&E 

The  importance  of  T&E  in  general  is  recognised  in  the  acquisition  process, 
to  identify  problem  areas,  to  allow  timely  corrective  action  to  be  taken,  and  to  reduce 
technical  risk.  As  Australia  has  a  policy  of  procuring  low  risk,  operationally  proven 
systems,  which  may  be  constructed  locally,  rather  than  higher  risk  local 
development,  the  RAN  concentrates  on  post  delivery  testing,  rather  than  testing  to 
support  a  production  or  purchase  decision.  The  importance  of  OT&E  to  the 
acquisition  process,  however,  is  not  adequately  appreciated.  The  RAN  appears  to 
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have  its  focus  on  PAT&E  to  meet  technical  and  fleet  standards,  rather  than  an 
assessing  operational  effectiveness  and  operational  suitability  against  the  user 
requirement. 

2.  OT&E  Policy 

The  lack  of  emphasis  given  to  OT&E  within  the  RAN  is  possibly  due  to  the 
lack  of  clear  guidance  on  OT&E  in  the  T&E  policy  documentation.  The  decision  to 
subject  an  acquisition  to  T&E  will  initially  be  decided  by  the  Sponsor  and  detailed 
in  the  Capability  Proposal,  however,  the  Sponsor  and  other  acquisition  managers 
require  clear  guidance  on  T&E  to  enable  them  to  make  effective  decisions.  The 
categories  of  ship  T&E  used  in  the  Australian  DoD  have  been  derived  from  the  U.S. 
DoD  instructions,  but  interpreted  to  accord  with  Australian  Requirements.  During 
this  interpretation  there  has  been  a  “blurring”  of  the  types  of  T&E,  resulting  in  a  lack 
of  clear  delineation  between  the  T&E  types.  This  problem  was  further  compounded 
by  the  adoption  of  the  PAT&E  based,  USN  Total  Ship  Test  Program  system,  and 
modifying  it  to  include  DT&E  and  OT&E  components. 

3.  RAN  OT&E  Organisation 

The  RAN  and  DoD  have  reorganised  over  the  past  five  years  to  create  a 
more  effective  and  efficient  user  requirements  organisation,  and  to  better  accord 
with  Program  Management  and  Budgeting  principles.  However,  as  a  result,  the 
RAN  has  no  T&E,  or  OT&E  policy  maker  and  overseer,  no  OT&E  coordinator 
within  Navy  Office,  and  the  RAN’s  OT&E  authority  is  buried  within  the  in-service 
engineering  management  area  of  the  Maritime  Command. 

4.  Understanding  of  OT&E 

As  a  result  of  incomplete,  contradictory  and  confusing  T&E  policy 
documentation,  emphasis  on  PAT&E  and  the  “blurring”  of  T&E  types,  the  lack  of 


true  OT&E  actually  conducted  and  few  personnel  experienced  in  OT«&E,  the 
understanding  of  OT«&E  within  the  RAN  is  poor.  As  a  consequence,  the  philosophy, 
methodology,  and  benefits  to  the  RAN  of  OT&E  are  not  adequately  recognised. 

5.  Summary 

The  demise  of  the  RAN’s  independent  OT&E  authority,  RANTAU.  and  its 
subsequent  integration  within  Maritime  Headquarters  as  the  Commander.  Test  and 
Evaluation  (GTE),  was  the  latest  in  a  series  of  reorganisations  that  has  significantly 
reduced  the  capability  and  authority  of  the  OT&E  community.  The  new  combatants 
currently  under  construction  for  the  RAN,  although  relatively  low  risk  by  USN 
standards,  are  a  higher  operational  risk  than  previous  RAN  ship  programs.  Although 
using  proven  technologies,  both  the  ANZAC  ships  and  Collins  class  submarines 
have  unique  weapon  and  sensor  fits,  not  currently  in  service  with  large  parent  navies. 
The  conduct  of  PAT&E  only,  is  not  considered  sufficient  to  measure  the  operational 
effectiveness  and  suitability  <rf  these  new  combatants  to  determine,  with  confidence, 
that  they  meet  the  user  requirement.  This  can  be  achieved  by  the  conduct  of  an 
effective  OT&E  program. 
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V.  USN/ RAN  OT&E  COMPARISON 


This  chapter  compares  the  characteristics  of  the  USN  and  RAN  OT&E  systems 
and  highlights  significant  differences. 

A.  DEFENCE  ACQUISITION  PROCESS 

The  Defence  system  acquisition  life  cycle  of  both  nations  consists  of 
progressive  development  phases  separated  by  major  decision  milestones. 
Milestones  are  periodic  formal  program  reviews  for  authorisation  to  advance  to  the 
next  stage.  The  two  systems  are  compared  in  Rgure  20.  With  a  policy  to  maintain 
world  leadership  in  key  Defence  technologies,  the  U.S.  places  priority  on 
development  programs.  The  major  decision  point  is  centred  on  the  decision  to 
proceed  beyond  Milestone  IIIA  -  Low  Rate  Initial  Production  ,  when  the  large 
dollars  in  an  acquisition  program  are  due  to  be  spent  in  the  production  phase. 
Passing  a  formal,  independent  OPEVAL  is  its  major  criterion  for  approval  to  Full 
Rate  Production. 

Although  the  decision  to  produce  or  purchase  a  new  system  is  important,  the 
Australian  DoD  prefers  lower  risk,  proven  technologies  for  the  majority  of  new 
Defence  acquisitions.  In  many  cases,  systems  with  the  fundamental  capabilities 
required  by  the  ADF  are  available  overseas,  however,  some  of  these  systems  require 
additional  development,  adaptation  or  integration  to  meet  Australian  specific 
requirements.  The  Australian  system,  therefore,  places  a  high  priority  on  the  post 
production  phase  with  the  three  milestones  of  Preliminary  Acceptance  (PA). 
Acceptance  into  Naval  Service  (AINS)  and  Operational  Acceptance  (OA). 
However,  Australia  is  now  buying  far  less  “off  the  shelf’  from  overseas,  and 
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systems  are  increasingly  being  designed  and  built  in  Australia,  or  at  least  adapted 
from  overseas  designs. 


U.S  PROCESS 

CONCEPT 

STUDIES 

approval 

CONCEPT 

DEMONSTRATION 

APPROVAL 

DEVELOPMENT 

APPROVAL 

PRODUCTION 

APPROVAL 

MAJOR 

MODIFICATION 

approval 

PHASE  0 

PHASE  1 

PHASE  II 

PHASE  III 

CONCEPT 

EXPLORATION 

AND 

DEFINITION 

DEMONSTRATION 

AND 

VALIDATION 

ENGINEERING  * 
MANUFACTURING 
DEVELOPMENT 

PRODUCTION 

AND 

DEPLOYMENT 

AUSTRALIAN  PROCESS 


FULL  SCALE 

PROGRAM  ENGINEERING  PRODUCTION 

INITIATION  DEVELOlWENT  APPROVAL 

APPROVAL 


AINS  OA 


OONCEPTUA 


PHASE  2 

PHASE  3 

PHAS  E  4 

PHASE  5 

DEMONSTRATION 

AND 

EVALUATION 

FULLSCALE 
ENGINEERING 
DEVE  LOPMENT 

PRODUCTION 

IN-SERVICE 

Figure  20:  Comparison  of  U.S.  and  Australian  Acquisition  Processes 


a  T&EDEnNITIONS 

Definitions  and  terminology  form  the  basis  of  a  T&E  system.  As  the  categories 
of  ship  T&E  used  in  the  Australian  DoD  have  been  derived  from  the  U.S.  DoD 
instructions,  both  Navies  recognise  three  types  of  T&E: 
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•  Developmental  Test  and  Evaluation  (DT&E) 

•  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

•  Operational  Test  and  Evaluation  (OT&E) 

The  categories  within  these  types  of  T&E,  however,  have  been  interpreted  by 
the  RAN  to  accord  with  Australian  requirements  [Ref.  38:p.  14-1].  This 
interpretation  leads  to  some  subtle,  but  significant,  differences  in  T&E  definitions 
between  the  two  Navies. 

1.  Developmental  Test  and  Evaluation  (DT&E) 

DT&E  is  conducted  to  assist  the  engineering  design  and  development 
process  and  to  verify  attainment  of  technical  performance  specifications  and 
objectives  [Ref.  1  l:p.  3].  Both  Navies  agree  that  DT&E  is  conducted  in  three  major 
phases.  Table  13  compares  these  DT&E  phases.  Minor  wording  differences  aside, 
the  three  DT&E  phases  are  essentially  identical.  The  most  significant  difference  is 
that  the  USN  recognises  PAT&E  as  a  form  of  DT-3  testing. 


TABLE  13;  COMPARISON  OF  DT&E  PHASES 


USN  Description 

DT 

RAN  Description 

Demonstration  that  all  technical  risk 
areas  have  been  identified  &  that  best 
technical  approaches  have  been 
accepted. 

DT-1 

Validation  of  design  concept. 

Demonstrate  that  design  meets 
specifications. 

DT-2 

Proving  design. 

Testing  conducted  on  production  items 
to  ensure  compliance  with  contracted 
specifications.  PAT&E  is  a  form  of 
DT.3. 

DT-3 

Demonstration  that  production  meets 
required  technical  characteristics. 
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2.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

Both  Navies  agree  that  PAT&E  is  testing  conducted  on  production  items  to 
ensure  systems  meet  contract  specifications  and  requirements.  PAT&E  for  ships  in 
the  USN  is  managed  under  the  Total  Ship  Test  Program  (TSTP)  which  includes 
seven  phases  of  PAT&E.  In  recent  years  the  RAN  has  adopted  the  TSTP,  initially 
with  the  modernisation  of  the  DDGs,  and  now  in  the  testing  of  the  locally  produced 
FFG-7  class  under  the  Australian  Frigate  Project.  The  adoption  of  the  TSTP  by  the 
RAN  included  the  redefinition  of  the  test  phases.  A  comparison  of  the  PAT&E 
phases  is  provided  in  Table  14.  There  are  major  differences  between  the  two  Navies 
in  these  PAT&E  categories.  The  USN  system  includes  only  PAT&E,  whereas  the 
RAN  includes  elements  of  DT&E  and  OT&E,  in  addition  to  PAT&E,  and  they  are 
given  PAT  designations. 

TABLE  14:  COMPARISON  OF  PAT&E  PHASES 


USN  Description 

PAT&E 

RAN  Description 

Not  defined  in  U.S.  system 

PAT-0 

Design  and  engineering  development 
tests 

Material  receipt  inspection  and  shop 
tests 

PAT-1 

Production  and  bum-in  tests 

Shipboard  installation  inspections 
and  tests 

PAT-2 

Environmental  qualification  tests 

Equipment  level  operational  tests 

PAT^ 

System  development  tests 

Intrasystem  tests 

PAT-4 

Harbour  testing 

Intersystem  tests 

PAT^ 

Sea  testing 

Special  tests  e.g.,  surveys,  rangings 

PAT-6 

Operational  and  Qualification  Trials 
(conducted  with  OT-3) 

Trials  tests  including  sea  trials, 
builders  trials,  acceptance  trials. 

PAT-7 

Follow  on  Operational  T&E 
(conducted  with  OT-4) 
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3.  Operational  Test  and  Evaluation  (OT&E) 

Both  Navies  concur  that  OT&E  is  conducted  to  determine  a  system’s 
operational  effectiveness  and  operational  suitability,  to  identify  system  deficiencies 
and  the  need  for  potential  modifications  to  meet  established  OT  thresholds,  and 
develop  tactics.  They  also  agree  that  OT&E  has  three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 

•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 

They  both  use  the  term  “Operational  Evaluation”,  abbreviated  as 
“OPEVAL”  to  cover  T&E  on  production  representative  baseline  equipment  using 
the  maintenance  and  support  personnel  and  equipment  for  normal  operational  use 
which  aims  to: 

•  Demonstrate  operational  effectiveness  and  suitability. 

•  Provide  data  to  assist  in  the  development  of  tactical  aspects  of  the  equipment. 

•  Verify  data,  handbooks  and  documentation  covering  the  operation  of  the 
system. 

The  USN  uses  the  term  OPEVAL  as  the  final  stage  of  OT-II,  supporting  a 
production  decision.  Although  the  Australian  definition  of  OT-2  includes  an 
OPEVAL,  the  term  is  not  used  exclusively  for  OT-2.  It  is  used  in  many  contexts 
from  OT-2  to  OT-4. 

There  are  also  significant  differences  in  the  detailed  OT&E  definitions.  The 
RAN  includes  ship  Certification  /  Qualification  trials  in  OT-3,  whereas  the  USN 
categorises  Certification  trials  as  PAT&E  and  Ship  Qualification  Trials  (SQT)  as 
regular  in-service  PAT&E  conducted  by  the  INSURV  Board  and  Type  Commander, 
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not  as  OT&E.  A  comparison  of  the  OT&E  phases  of  the  USN  and  RAN  are  given  in 
Table  15. 


TABLE  15:  COMPARISON  OF  OT&E  PHASES 


USN  Description 

OT 

RAN  Description 

Early  Opjeraiional  Assessment 

OT-1 

Operational  assessment  of  the 
development  proposal 

OT&E  conducted  to  support  a 
production  decision.  Final  phase  is 
termed  OPEVAL. 

OT-2 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  suitability  of  a 
prototype  (OPEVAL)  to  support 
proceeding  to  full  production 

OT&E  conducted  on  production 
system  to  verify  correction  of 
deficiencies  after  OPEVAL,  and 
certification  of  operational 
effectiveness  and  suitability. 

OT-3 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  suitability  on 
production  of  ship/aircraft/system, 
normally  in  independent  operations 
using  normal  Fleet  personnel. 
Certincation/Qualification  Trials 
include  limited  reliability, 
maintainability,  availability  and 
logistic  supportability  assessments. 

Validation  of  the  operational 
effectiveness  and  suitability  of 
production  systems.  Usually  in 
different  environments  and  to  assess 
integrated  operation  of  system.  For 
ship  programs,  is  normally  conducted 
to  verify  that  critical  deficiencies  have 
been  identified  and  to  complete  any 
outstanding  OT&E. 

OT4 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  operational 
suitability  on  production  of  ship/ 
aircraft/system  using  normal  Fleet 
personnel  in  a  multi-force,  multi-threat 
environment,  Includes  detailed 
reliability,  maintainability,  availability 
and  logistic  supportability 
assessments. 

Not  defined  in  the  U.S.  System. 

OT^ 

Follow-on  OT  for  assessment  after 
modernisations,  new  applications  or 
defect  rectifications  after  OT-4 
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C.  T&E  IN  THE  ACQUISITION  PROCESS 

Both  Navies  appreciate  the  importance  of  T&E  in  the  acquisition  process.  T&E 
results  are  assessed  as  part  of  the  milestone  reviews.  However,  the  level  of 
importance  of  T&E,  and  OT&E  in  particular,  differs  greatly. 

1.  The  Requirement  for  T&E 

The  importance  of  T&E,  and  OT&E  in  particular,  to  the  progressive 
assessment  of  programs  in  the  USN  is  such  that  its  use  has  been  mandated  by  the 
U.S.  Congress  and  incorporated  into  the  laws  of  the  United  States.  The  law  includes 
the  provision  of  a  Director  of  Operational  Test  and  Evaluation  (DOT&E)  in  the 
Department  of  Defence.  The  law  also  addresses  specific  areas  of  OT&E  reporting 
and  conduct  to  ensure  the  Congress  is  kept  informed,  and  the  testing  and  reporting 
are  impartial  [Ref.  17].  By  including  the  requirement  that  a  major  Defence 
acquisition  program  may  not  proceed  beyond  low-rate  initial  production  until  initial 
OT&E  of  the  program  is  completed.  Congress  ensures  that  OT&E  is  an  integral  part 
of  the  acquisition  process. 

The  USN  appreciates  that,  if  adequate  OT&E  is  not  done,  and  the  weapon 
system  does  not  perform  satisfactorily  in  the  field,  significant  changes  may  be 
required.  Moreover,  the  changes  will  not  be  limited  to  a  few  developmental  models, 
but  may  also  be  applied  to  items  already  produced  and  deployed.  The  USN  also 
recognises  that,  in  extreme  situations,  it  also  risks  deploying  systems  which  cannot 
adequately  perform  significant  portions  of  their  missions,  thus  degrading  its 
deterrent  /  defensive  capabilities  and  endangering  the  safety  of  military  personnel 
who  operate  and  maintain  the  systems. 

The  need  for  local  T&E  is  recognised  by  the  Australian  Government  and 
DoD,  for  both  indigenous  and  overseas  systems,  however,  the  importance  of  OT&E 
in  particular,  is  not  specifically  addressed  in  any  Australian  T&E  policy  document. 
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Perhaps  this  lack  of  recognition  of  the  importance  of  OT&E  is  due  to  the  RAN 
procuring  low  risk,  complete  ships  in  the  past  (e.g.,  FFG-7  class)  with  the  wealth  of 
USN  DT&E  and  OT&E  behind  it.  Also  the  RAN,  unlike  the  USN,  has  not 
experienced  the  operational  situations  in  recent  years  which  may  have  highlighted 
operational  effectiveness  and  suitability  deficiencies  in  its  ships.  With  the  RAN  now 
buying  higher  risk,  unproven  combinations  of  systems  within  its  ships,  it  must  now 
rely  on  its  own  OT&E. 

2.  T&E  Contributions  At  Major  Milestones 

The  USN  OT&E  system  forms  an  integral  part  of  the  acquisition  process. 
Program  advance  from  one  phase  to  the  next  is  not  by  the  calendar  of  planned 
schedule,  but  by  actual  resolution  of  critical  operational  issues  and  achievement  of 
pre-set  thresholds  verified  by  T&E  [Ref.  1  l:p.  2].  OT&E  in  the  RAN,  on  the  other 
hand,  may  be  conducted  during  each  phase  in  the  acquisition  process,  but  it  is  not 
essential  to  a  local  system  meeting  a  development  or  production  milestone,  or  the 
local  production  or  purchase  decision  of  an  overseas  system.  It  is  however,  required 
before  a  system  achieves  Acceptance  into  Naval  Service  or  Operational  Acceptance. 
Figure  21  compares  the  contributions  of  T&E  to  the  acquisition  milestones  for  each 
Navy. 

Although  the  diagrams  appear  to  be  very  similar,  the  RAN  OT&E 
contributions  to  the  acquisition  milestones  are  by  no  means  definitive.  There  is  no 
documented  requirement  for  the  conduct  of  OT-1  and  /  or  OT-2  prior  to  a  production 
or  purchase  decision  of  major  acquisitions.  The  decision  to  conduct  OT&E  being  the 
responsibility  of  the  Sponsor,  Project  Manager  and  the  Defence  acquisition 
committees.  RAN  T&E  policy  documentation  almost  exclusively  focuses  on  the 
post  production  acceptance  phases.  Even  during  the  acceptance  phase,  however,  the 


requirement  for  OT-3  is  clouded  in  definition  differences  between  policy  documents. 
The  RAN  appears  to  have  its  focus  on  PAT&E. 
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Figure  21:  Comparison  of  T&E  Contributions  to  Acquisition  Milestones 


Of  note  is  that  the  USN  diagram  shows  separate  activities  for  DT«feE  and 
OT&E  related  tests,  while  the  RAN  diagram  groups  them  all  together.  This 
demonstrates  the  delineation  between  DT&E  /PAT«feE  and  OT&E  evident  in  the 
USN  system. 

D.  OT&E  ORGANISATION 

The  OT&E  organisation  consists  of  a  number  of  functions.  The  authorities 
performing  these  functions  in  each  Navy  are  listed  in  Table  16. 

TABLE  16:  COMPARISON  OF  OT&E  AUTHORITIES 


USN 

Authority 

OT&E 

Function 

RAN 

Authority 

Director, 

Policy  Maker 

Operational  Test  &  Evaluation 
(DoD) 

&  Overseer 

Nil 

N8  -  Warfare  Requirements 

Sponsor 

Director  General  Force 

Chief  of  Naval  Operations 

Development  (Sea) 

Commander, 

Developing 

Assistant  Chief  of  Naval  Staff  - 

Naval  Sea  Systems  Command 

Agency 

Materiel 

Commander,  Operational 

OT&E  Tester 

Commander,  Test  &  Evaluation 

Test  and  Evaluation  Force 

&  Evaluator 

Ships  Commanding  Officers, 

User 

Ships  Commanding  Officers, 

Reet  Commanders 

Maritime  Commander 

Director, 

OT&E 

Test,  Evaluation  & 

Technology  Requirements 

Coordinator 

Nil 
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1.  OT&E  Policy  Maker  and  Overseer 

An  OT&E  policy  maker  and  overseer  is  employed  in  the  U.S.  system 
responsible  for  policy  formulation,  evaluation  and  oversight  of  ail  OT&E.  The 
Director  of  OT&  E  (DOT&E)  is  the  principal  OT&E  official  in  the  Department  of 
Defence  and  the  principal  advisor  to  the  Secretary  of  Defence  on  OT&E.  In  addition, 
a  Director  T&E,  responsible  for  developmental  testing  policy  reports,  to  the 
USD(A)  in  DoD. 

There  is  no  clear  OT&E,  or  T&E,  policy  maker  or  overseer  within  the 
Australian  DoD  or  the  RAN.  The  Chief  of  Capital  Procurement  [Ref.  38]  addresses 
T&E  in  general,  but  more  in  the  form  of  guidance  rather  than  policy.  The  former 
Director  General  of  Naval  Warfare  (DGNW)  drafted  the  RAN  Test  and  Evaluation 
Policy  Defence  Instruction  [Ref.  44],  however  after  the  recent  reorganisation.  T&E 
policy  for  the  RAN  is  presently  in  a  state  of  flux,  with  no  office  having  assumed  the 
responsibility  [Ref.  50]. 

2.  Program  Sponsor 

The  program  sponsor  is  responsible  for  the  development  of  concepts  for 
operations  for  naval  warfare  systems,  and  for  acquisition  program  requirements  and 
related  system  thresholds.  He/She  is  also  responsible  for  the  subsequent  trials  and 
continuing  overview  to  ensure  that  the  equipment  meets  the  requirement.  Within  the 
USN,  major  warship  programs  are  sponsored  by  N8  -  Warfare  Requirements,  under 
CNO.  Under  the  new  USN  organisation,  increasing  importance  is  given  to  the  Fleet 
and  Type  Commanders  in  determining  requirements  and  allocating  priorities. 
Besides  generating  the  user  requirement,  the  sponsor  also  has  a  review  function  in 
the  OT&E  process.  [Ref.  20] 

The  Australian  Defence  Force  has  a  centralised  force  development  and  user 
requirements  organisation,  within  HQADF,  the  Director  General  Force 


Development  (Sea)  having  sponsorship  of  major  RAN  acquisitions.  The  decision  to 
subject  an  acquisition  to  T&E  in  the  RAN  is  initially  decided  by  the  sponsor  and 
detailed  in  the  capability  proposal  [Ref.  44:p.  1]. 

3.  Developing  Agency 

When  a  program  is  approved,  a  program  office  is  formed  to  manage  the 
acquisition  of  the  system.  In  the  case  of  the  USN,  this  office  is  established  within  the 
Naval  Sea  Systems  Command  (NAVSEA).  The  program  office  is  responsible  for  the 
conduct  of  DT&E  in  preparation  for  OT«feE.  To  assist  all  NAVSEA  programs  with 
T&E  matters,  NAVSEA  have  established  their  own  T&E  (rffice  (SEA  60).  Although 
primarily  a  DT&E  policy  authority,  this  office  also  has  an  input  into  the  OT&E 
process  by  reviewing  the  TEMP,  and  by  conducting  OT&E  readiness  reviews  of 
NAVSEA  programs. 

Within  the  RAN,  the  Materiel  Division  is  responsible  for  acquisition  of  the 
capital  equipment  to  meet  the  requirement.  Similar  to  the  USN,  the  nature  and  extent 
of  the  T&E  to  be  conducted  is  decided  by  the  Project  Director  in  consultation  with 
the  Design  Approval  Authority  and  RANTAU.  However,  the  Materiel  Division  does 
not  have  a  T&E  office  to  support  Project  Directors. 

4.  OT&E  Tester  and  Evaluator 

The  Operational  Test  and  Evaluation  Force  (OPTEVFOR)  is  the  USN’s 
sole  independent  test  agency  responsible  for  the  planning  and  conduct  of  OT&E. 
COMOPTEVFOR  reports  directly  to  the  CNO,  and  is  separate  and  distinct  from  the 
developing  and  procuring  commands. 

The  RAN  Trials  and  Assessing  Unit  (RANTAU)  was  the  RAN's  OT&E 
authority  who  also  conducted  DT&E  /  PAT&E  on  behalf  of  Project  Directors  and 
Ship  (Qualification  Trials  for  the  Maritime  Commander.  With  the  disestablishment  of 
RANTAU  in  May  1992,  these  trials  functions  were  transferred  to  the  Commander. 
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Test  and  Evaluation  (CTE)  under  the  Chief  Staff  Officer  (Engineering)  in  the 
Maritime  Command. 

5.  User 

The  Commanding  Officer  and  crew  of  the  ship  under  test,  and  his  Fleet  and 
Type  Commanders  are  the  ultimate  users  of  the  system.  The  Maritime  Commander, 
and  the  individual  ships'  Commanding  Officers  and  crews  are  the  ultimate  users  of 
a  ship.  The  personal  assessment  of  the  Commanding  Officer  of  the  ship  under  test  is 
sought  by  OPTEVFOR  during  the  conduct  of  OT&E  [Ref.  15:p.  44].  The 
Commanding  Officer  of  a  ship  under  test  usually  provides  CTE  with  his  opinions  on 
the  ship’s  performance  at  the  conclusion  of  a  trials  period. 

6.  Coordinator 

CNO  has  responsibility  for  ensuring  the  adequacy  of  the  Navy's  overall  test 
and  evaluation  program.  T&E  policy  and  guidance  are  exercised  through  the 
Director,  Test,  Evaluation  and  Technology  Requirements  (N091).  This  organisation 
also  acts  as  a  T&E  Focal  Point  and  Coordinator,  responsible  for  coordination  of 
T&E  matters  in  the  designated  Progiams,  System  Commands  and  Department  of  the 
Navy,  [Ref.  13:p.  7-5]He  also  chairs  the  Test  and  Evaluation  Coordination  Group 
The  RAN  currently  has  no  authority  with  this  role,  however,  the  T&E  manager 
within  major  projects  is  responsible  for  chairing  the  Test  and  Evaluation  Planning 
Group. 

7.  OT&E  Organisation  Summary 

A  comparison  cf  the  OT&E  organisation  structures  of  both  Navies  is  shown 
in  Figure  22.  The  USN  OT&E  organisation  is  well  structured  with  comprehensive 
responsibilities  for  each  participant,  and  clearly  delineated  from  the  DT&E 
structure.  COMOPTEVFOR  is  the  truly  independent  tester  and  evaluator,  being 


separate  from  developing,  production  and  user  agencies,  and  responsible  only  for 
OT&E.  As  a  Rear  Admiral  and  reporting  directly  to  the  CNO.  he  has  authority, 
visibility  and  influence.  The  office  of  DOT&E  provides  a  layer  of  oversight  on  the 
OT&E  process  to  ensure  objective  testing  is  conducted,  giving  credible  results  and 
leading  to  impartial  decisions.  NAVSEA  has  its  own  T&E  agency  to  assist  PM’s  in 
DT&E,  and  in  preparation  for  OT&E.  The  OT&E  process  is  coordinated  from 
within  CNO’s  office  by  the  Director,  Test,  Evaluation  and  Technology 
Requirements. 

Following  a  number  of  recent  reorganisations,  the  RAN's  OT&E  structure 
is  in  disarray.  The  RAN’s  OT&E  authority  is  now  CTE  within  the  Maritime 
Command.  In  addition  to  OT&E  responsibilities,  he  also  conducts  some  DT&E.  and 
some  functions  equivalent  to  the  USN  INSURV  board,  OPTEVFOR  and  the  Type 
Commander.  Only  of  Commander  rank,  he  lacks  authority,  particularly  with  Project 
Directors  and,  being  within  the  in-service  engineering  area,  he  possibly  lacks 
credibility  and  influence  on  operational  matters  pertaining  to  OT&E.  In  developing 
programs,  the  RAN  relies  on  the  minimal  interface  between  the  centralised  sponsor 
(DGFD(Sea))  and  CTE  to  ensure  appix^riate  OT&E  is  considered,  since  the  sponsor 
has  no  agency  within  his  own  organisation  to  provide  the  necessary  advice.  Once  a 
project  is  established,  the  Project  Director  performs  the  coordinating  role  through 
the  TEPG.  The  Materiel  Command,  however,  has  no  T&E  agency  to  advise  PD’s  on 
this  role,  and  to  be  the  retainers  of  “lessons  learned”  from  other  projects.  Finally, 
with  no  clear  policy  maker  for  T&E  in  general,  or  OT&E  in  particular,  the  RAN 
lacks  the  ability  to  effectively  manage  overall  T&E,  let  alone  OT&E,  within  the 
RAN. 
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Figure  22:  OT&E  Organisation  Comparison 
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E.  OT&E  POLICY 


USN  OT&E  policy  has  its  basis  in  statutes  approved  by  Congress.  The  RAN  has 
no  such  formal  basis  and  its  OT&E  policy  is  embedded  in  the  more  general  T&E 
policy.  A  number  of  OT&E  policies  of  interest  include: 

1.  OT&E  Documentation 

USN  policy  is  refined  in  the  DoD  acquisition  Directives  5000.1  [Ref.  25] 
and  5000.2  [Ref.  24].  The  essential  T&E  elements  in  these  directives  are  further 
detailed  in  OPNAV  Instruction  3960. IOC  [Ref.  11]  and  other  subordinate 
instructions.  USN  OT&E  documentation  is  comprehensive  and  provides  clear, 
consistent  and  non-conflicting  policies  and  guidance  to  OT&E  participants.  This 
concise  documentation  establishes  the  disciplined  management  approach  to  OT&E 
taken  by  the  USN. 

The  RAN  does  not  enjoy  this  sound  basis  of  OT&E  policy.  Some  general 
guidance  to  assist  Project  Directors  in  assessing  the  need  for  T&E  in  a  project  and 
to  provide  guidelines  for  its  conduct  is  provided  in  the  Capital  Equipment 
Procurement  Manual  (CEPMAN  l)[Ref.  38].  The  top  level  T&E  policy  document 
for  the  RAN  is  the  Defence  Instruction  (Navy)  on  RAN  Test  and  Evaluation  Policy 
[Ref.  44],  while  ABR  1921  [Ref.  41],  contains  T&E  and  acceptance  policy 
applicable  to  ships  building,  undergoing  modernisation,  conversion  or  extended 
refit.  These  documents  provide  broad  T&E  policy  only  and,  other  than  defining 
OT&E,  provide  little  policy  as  to  its  conduct. 

2.  OT&E  Requirements 

In  the  USN  system,  the  requirement  to  conduct  OT&E  on  new  major 
acquisitions  is  mandatory.  Although  the  Sponsor  initially  develops  the  system 
thresholds,  the  level  and  extent  of  OT&E  required  is  delineated  in  the  TEMP  and 
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approved  prior  to  Milestone  I.  The  appropriate  level  of  OT&E  is  determined  by 
COMOPTEVFOR  in  consultation  with  the  Program  Manager,  with  DOT«feE 
oversight.  This  ensures  that  OT&E  planning  is  conducted  and  approved  early  so  the 
costs  of  OT&E  can  be  included  in  project  funds. 

In  the  RAN,  the  decision  to  subject  an  acquisition  to  T&E  is  initially 
decided  by  the  Sponsor  and  detailed  in  the  Capability  Proposal.  The  nature  and 
extent  of  the  T&E  (no  distinction  is  made  in  RAN  policy  between  the  various  types 
of  T&E)  is  decided  by  the  Project  Director  in  consultation  with  the  Design  Approval 
Authority  and  CTE.  Any  conflict  over  the  requirement,  nature  and/or  extent  of  the 
proposed  T&E  is  resolved  by  DCNS  and  ACMAT-N  for  operational  and  design 
matters  respectively. [Ref.  44:p.  2]  The  role  of  DCNS  should  now  rest  with 
DGFD(Sea)  as  the  Sponsor.  The  determination  of  the  OT&E  required  has 
traditionally  not  been  achieved  early  enough  in  the  life  of  a  project,  and  so  project 
funding  has  not  included  the  provision  for  OT&E. 

3.  Independence  of  OT&E  Authority 

As  OT&E  is  the  final  exam  for  an  acquisition  program,  independence  is 
necessary  so  that  there  is  no  question  of  impropriety.  The  USN  requires  its 
independent  operational  test  activity  to  be  separate  and  independent  from  the 
materiel-developing  and  procuring  agency  and  the  using  agency.  This  ensures  that 
the  testing  and  reporting  agency  has  nothing  to  gain  or  lose  by  whatever  they  say.  As 
major  acquisition  decisions  hinge  on  the  results  of  OT&E  within  the  USN, 
independence  is  necessary  to  ensure  that  the  reports  on  which  those  decisions  are 
based  are  not  self  serving.  [Ref.  19] 

Although  major  acquisition  decisions  within  the  RAN  are  less  based  on 
OT&E,  the  RAN  also  maintains  the  policy  that  OT&E  is  to  be  conducted  by  an 
authority  independent  of  the  development  and  production  agencies.  With  the  demise 
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of  RANTAU  and  the  transfer  of  OT&E  responsibilities  to  CTE  on  the  staff  of  the 
Maritime  Commander,  independence  from  the  development  and  production 
agencies  is  achieved,  but  not  from  the  end  user  as  in  the  USN.  The  implications  of 
this  lack  of  independence  are  that  the  end  user  may  have  his  own  aims  and  objectives 
which  may  conflict  with  the  total  impartiality  of  OT&E  reporting.  The  politics  of 
requirements  and  funding  could  come  into  play,  as  the  Maritime  Commander 
establishes  scheduling  priorities  for  his  resources,  and  OT&E  may  take  a  low 
priority  behind  operations  and  training.  The  Maritime  Commander  may  also  not 
wish  to  “stress”  the  system,  in  case  he  perceives  that  operational  deflciencies  so 
highlighted  may  be  seen  as  having  some  reflection  on  his  own  organisation. 

4.  Contractor  Involvement  in  OT&E 

To  ensure  impartiality  and  credibility  of  testing,  the  USN  prohibits  persons 
employed  by  the  contractor  for  the  system  being  tested  being  involved  in  OT&E, 
unless  contractors  are  planned  to  be  involved  when  the  system  is  deployed  in 
combat. 

The  RAN  requires  that  during  OT&E,  a  system  should  be  demonstrated 
“using  normal  Fleet  Personnel”,  however,  the  policy  documentation  makes  no 
statement  regarding  contractor  personnel  involvement  in  system  operation  or 
maintenance  during  OT&E.  It  has  been  RAN  practice  (e.g.,  in  the  DDG 
Modernisation  Project)  to  have  contractors  onboard  resolving  technical  problems 
during  the  SQT  period,  which  is  included  as  part  of  OT-3  in  the  RAN,  thereby 
possibly  causing  test  bias. 

5.  T&E  Planning 

Both  the  USN  and  RAN  use  the  Test  and  Evaluation  Master  Plan  (TEMP) 
as  the  single,  executive,  long  range  planning  document  for  T&E  in  a  project.  Both 
Navies  are  in  agreement  that  approval  of  the  TEMP  constitutes  establishment  of  a 


contract  between  the  requirement-setter  and  the  PM  which  is  essential  to  formal  and 
accountable  program  execution,  and  between  the  requirement-setter  and  the  OT 
agency,  upon  which  the  OT  agency  will  independently  evaluate  system  operational 
effectiveness  and  suitability.  The  main  differences  between  the  two  Navies  are  in  the 
responsibility  and  level  of  approval,  and  in  the  timing  of  the  TEMP. 

a.  TEMP  Responsibility 

Within  the  USN,  the  TEMP  is  drafted  by  the  PM.  COMOPTEVFOR 
drafts  Part  Four  on  OT&E  and  provides  OT&E  resource  requirements  for  Part  Five. 
The  TEMP  is  then  co-submitted  by  both  the  Program  Manager  and 
COMOPTEVFOR.  It  is  reviewed  by  the  Assistant  Chief  of  Naval  Operations 
(Surface  Warfare),  the  Director,  T&E  and  Technology  Requirements,  and  the 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition).  It  is 
ultimately  approved  at  the  DoD  level  by  the  Director,  OT&E  and  the  Director,  Test 
and  Evaluation. 

Within  the  RAN,  the  TEMP  is  the  ultimate  responsibility  of  the  project 
sponsor  [Ref,  44;p.  4].  However,  during  the  period  when  there  is  a  full  time  Project 
Office,  TEMP  coordination  is  normally  carried  out  by  the  T&E  Manager  in  the 
Project  Office.  CTE  as  the  OT&E  authority  provides  the  OT&E  input.  The  TEMP 
is  endorsed  by  all  affected  organisations  and  is  released  jointly  by  the  Deputy  Chief 
of  Naval  Staff  (DCNS),  and  the  Assistant  Chief  of  Naval  Staff- Materiel  (ACMAT- 
N),  as  the  Developing  Agency.  If  any  issues  remain  unresolved  at  this  level,  the 
TEMP  will  be  resolved  by  the  Chief  of  Naval  Staff  (CNS).  [Ref.  44;pp.  4-5]  The 
authority  for  DCNS  to  release  the  TEMP  was  based  on  his  role  as  Project  Sponsor. 
With  HQADF  now  assuming  this  role,  the  DCNS  function  in  TEMP  development 
is  unclear. 
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In  the  USN,  drafting  the  TEMP  is  the  responsibility  of  the  PM  while  the 
DoD  heads  of  OT&E  and  DT&E  authorities  approve  the  TEMP.  In  contrast,  in  the 
RAN,  the  Sponsor  has  ultimate  responsibility,  although  the  TEMP  is  managed  by 
the  PM,  and  the  TEMP  is  approved  by  the  heads  of  the  developing  agency  and  the 
(former)  Sponsor. 

b.  TEMP  Timing 

The  TEMP  is  required  in  the  at  Milestone  I-  Concept  Demonstration 
Approval  within  the  USN,  and  is  updated  for  each  subsequent  milestone  and  when 
significant  program  changes  occur.  The  RAN  requires  the  TEMP  to  be  raised  in 
draft  form  as  early  as  practicable,  but  in  actuality  the  TEMP  for  major  projects  is 
usually  drafted  after  production  approval  (Milestone  3). 

6.  Combined  DT&E  and  OT&  E 

“Combined  testing”  refers  to  a  single  test  program  conducted  to  support 
both  DT&E  and  OT&E  objectives.  The  advantages  of  combined  testing  is  the 
shorter  time  required  for  testing,  and  cost  savings  by  eliminating  redundant 
activities.  These  need  to  be  weighed  against  the  limitations  of  the  additional 
extensive  coordination  required  and  the  less  than  optimum  environment  and 
coverage  for  OT&E  that  may  occur.  Early  involvement  of  OT&E  personnel  during 
system  development  increases  their  familiarity  with  the  system  and  permits 
identification  of  operational  concerns  early  in  the  program. 

The  USN  has  a  policy  that  combined  DT&E  and  OT&E  testing  should  be 
considered  when  there  are  time  and  cost  savings  [Ref.  24:p.  8-3].  However,  this 
combined  approach  must  not  compromise  either  the  developmental  or  operational 
test  objectives.  Also  a  final  independent  phase  of  operational  testing  and  evaluation 
is  required  for  beyond  low  rate  initial  production  decisions  [Ref.  24:p.  8-4].  The  US 
has  extensive  experience  with  combined  testing,  some  successful  and  some 
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unsuccessful,  and  they  conclude  that  it  is  possible  to  have  combined  test  teams 
involved  throughout  the  testing  process.  The  DT&E  and  OT&E  teams  can  share 
mutually  beneficial  data,  as  long  as  the  test  program  is  carefully  planned.  However, 
evaluation  and  reporting  activities  must  be  conducted  separately.  [Ref.  18:pp.  17-1 
-  17-4]. 

The  conduct  of  combined  DT&E,  (PAT&E)  and  OT&E  is  not  addressed  in 
RAN  T&E  policy,  however,  it  is  implied  as  being  almost  a  requirement.  The  T&E 
DI(N)  [Ref.  44:p.  3]  states  that  OT-3  and  OT-4  will  incorporate  the  requirements  of 
PAT-6  and  PAT-7  Inspection  Tests  and  Trials  as  described  in  ABR  1921.  It  also  states 
that  OT-3  will  be  conducted  during  Ship  Qualification  T rials.  The  RANs  “blurring" 
of  the  distinction  between  the  types  of  T&E  leads  to  DT&E  and  OT&E  often  being 
conducted  over  the  same  period,  by  the  same  test  team.  Although  the  RAN 
recognises,  by  definition,  the  difference  in  objectives  and  methodology  between  the 
types  of  T&E,  no  limitations  or  guidance  as  to  the  possible  hazards  of  this  combined 
testing  approach  are  addressed. 

7.  Modeling  and  Simulation 

The  use  of  modeling  and  simulation  can  increase  the  efficiency  of  the  T&E 
process,  reduce  the  time  and  cost,  provide  otherwise  unattainable  and 
unmeasureable  data,  and  provide  more  timely  and  valid  results.  The  USN 
encourages  the  use  of  simulation  and  modeling  to  assist  in  projecting  operational 
effectiveness  and  operational  suitability  prior  to  Milestone  II  -  Development 
Approval,  but  limit  the  use  of  simulation  and  modeling  in  subsequent  OT&E  to  that 
of  supplementing  OT&E  test  data  [Ref.  15:p.  2-37]. 

The  use  of  models  and  simulation  in  OT&E  is  not  addressed  by  the  RAN 
documentation,  although  modeling  and  simulation  performed  by  the  Maritime 
Systems  Division  of  the  Defence  Science  and  Technology  Organisation  were  used 
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by  RANTAU  to  assist  the  evaluation  of  the  upgraded  combat  and  weapons  systems 
in  the  Modernised  DDGs. 

With  the  increasing  costs  of  weapons  systems,  decreasing  budgets  and  the 
subsequent  need  for  efficiency  in  the  conduct  of  OT&E,  simulation  can  be  used 
during  OT&E  to  extend  test  results,  to  improve  the  statistical  sample,  or  to 
determine  overlooked  or  directly  unmeasured  parameters.  This  is  particularly 
relevant  to  the  RAN  who  lack  appropriate  ranges  and  the  financial  resources  for 
extensive  test  programs  of  expensive  weapons.  A  policy  directing  the  use  of  models 
and  simulation  in  OT&E  in  the  RAN  is  required. 

8.  Foreign  Weapons  Evaluation 

The  US  DoD  has  a  foreign  weapons  evaluation  program  which  is  designed 
to  support  the  evaluation  of  a  foreign  nation's  weapons  system,  equipment  or 
technology  in  terms  of  its  potential  to  meet  a  specific  U.S.  military  requirement  The 
primary  objective  of  the  program  is  to  reduce  the  costs  of  research  and  development, 
while  leading  to  the  acquisition  of  foreign  equipment  for  U.S.  use  [Ref.  18:p.  21-1]. 
From  the  OT&E  viewpoint,  the  USN  ‘try-before-buy  policy’  is  still  maintained, 
despite  the  early  phases  of  OT  being  unable  to  be  achieved.  When  procurement  of  a 
foreign  weapon  system  is  planned,  the  developing  agency  and  COMOPTEVFOR 
may  be  directed  to  assess  the  adequacy  of  any  previously  conducted  DT&E  and 
OT&E  and  to  provide  recommendations  on  the  need  for  additional  T&E  prior  to 
procurement  [Ref.  15:p.  2-34]. 

Despite  the  most  recent  Australian  Government  “White  Paper’’  on  Defence 
Policy  [Ref.  6:p.  70]  requiring  the  need  to  be  able  to  determine  the  performance  in 
the  Australian  environment  of  equipment  of  both  overseas  and  local  origin,  no  DoD 
or  RAN  T&E  policy  specifically  addresses  the  evaluation  of  foreign  systems.  Prior 
to  a  major  acquisition,  project  teams  review  possible  contractors’  proposals  and 
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evaluate  them  against  the  user  requirement.  OT&E  personnel  may  be  part  of  these 
teams,  however,  there  is  no  policy  as  to  the  OT&E  authority’s  role  in  foreign 
weapons  evaluation  within  the  RAN. 

9.  OT&E  Funding 

OT&E  is  an  expensive  process  and  generally  occurs  late  in  the  life  of  a 
project  when  money  is  usually  tight.  Within  the  USN,  funding  associated  with  T&E 
(including  instrumentation,  targets  and  simulations)  are  identified  in  the  system 
acquisition  cost  estimates,  acquisition  plans  and  the  TEMP  [Ref.  18:p.  18-11].  The 
Program  Manager  plans,  programs,  budgets  and  funds  the  costs  of  all  resources 
identified  in  the  approved  TEMP  for  all  T&E  through  OT-III.  Funds  required  to 
conduct  OT&E  are  programmed  and  budgeted  by  OPTEVFOR  and  advised  to  the 
Program  Manager.  OT&E  costs  include  test  articles,  expendables,  targets,  data 
collection  and  reduction  and  OPTEVFOR  program  related  costs.  The  Program 
Manager  does  not  fund  fleet  operating  costs  for  T&E  support,  which  includes  fuel 
and  aircraft.  These  costs  for  OT-II  and  III,  and  all  costs  for  OT-IV.  except 
procurement  costs  and  OPTEVFOR  costs,  are  funded  by  the  Fleet  CINCs.[Ref. 
1 1  :p.  21]  The  RAN  employs  a  similar  funding  arrangement  where  OT&E  is  funded 
separately  within  a  project  and  is  estimated  and  bid  for  by  CTE.[Ref.  44:p.  6] 

The  essential  difference  is  that  the  USN  develops  the  TEMP,  and  hence 
identify  the  resources  required  and  their  costs,  earlier  in  a  project’s  life  than  does  the 
RAN,  thus  enabling  the  project  to  include  these  requirements  in  project  funding. 

10.  Land  Based  Test  Sites 

A  Land  Based  Test  Site  (LBTS)  is  a  facility  that  duplicates,  simulates  or 
stimulates  the  employment  of  a  system's  planned  operational  installation  and 
utilisation.  Used  primarily  for  the  purpose  of  conducting  DT&E,  it  is  sometimes 
used  to  test  system  integration  and  overall  performance.  The  USN  often  use  LBTS 
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for  the  conduct  of  initial  OT&E  to  gain  an  estimation  of  potential  operational 
effectiveness  and  suitability,  primarily  to  support  Milestone  IIIA  (Low  Rate  Initial 
Production)  decisions  and  not  the  more  stringent  Milestone  IIIB  (Full  Rate 
Production). 

RAN  guidance  for  OT&E  using  land  based  test  sites  includes  a  statement 
that  system  centres  and  simulators  will  be  employed  for  early  stages  of  OT&E  if 
available  p[^ef.  44:p.  4],  and  that  OT-2  may  be  conducted  at  a  land-based  test  site 
[Ref.  44:p.  3].  Although  LBTS  are  being  developed  for  both  the  ANZAC  frigate  and 
COLLINS  submarine  projects,  OT-2  in  support  of  production  decisions  will  not  be 
conducted.  The  use  of  these  LBTS  for  later  OT&E  was  not  planned  in  the  original 
project  schedules,  and  so  is  subject  to  them  being  used  on  a  non-interfering  basis  to 
the  contractor. 

The  USN  has  demonstrated  that  early  OT&E  on  LBTS  can  give  an 
estimation  of  potential  operational  effectiveness  and  suitability,  and  hence  identify 
potential  operational  problems  early  and  minimising  operational  risk.  Their  use  for 
OT&E  during  the  ANZAC  frigate  and  COLLINS  submarine  projects  would  appear 
to  offer  similar  advantages  to  the  R/iN. 

11.  Ship  OT&E 

Both  the  USN  and  RAN  recognise  that  ship  acquisition  projects  will  be 
treated  differently  to  others  in  so  far  as  protracted  construction  time  usually 
precludes  T&E  of  a  prototype  before  the  decision  to  proceed  to  production.  Because 
the  development  and  construction  period  for  a  major  ship  in  the  USN,  normally 
precludes  completion  of  initial  OT&E  on  the  lead  ship  prior  to  the  production 
decision  for  follow-on  ships,  successive  phases  of  OT&E  are  accomplished  as  soon 
as  practicable  to  reduce  risk  and  minimise  the  need  for  modification  to  follow-on 
units.  Ship  acquisition  programs,  therefore,  usually  have  OT&E  between  Milestones 


II  and  III,  which  consist  of  individual  weapon  systems  testing  and  system  integration 
at  land  based  test  sites. 

With  the  construction  of  lower  risk,  proven  overseas  ships  adapted  to 
Australian  requirements,  the  RAN  assessed  the  degree  of  technical  risk  as  low. 
OT«feE  between  Milestones  II  and  III  in  the  RAN  was,  therefore,  not  considered 
necessary. 

F.  SHIP  ACCEPTANCE 

Both  Navies  have  similar  policies  and  procedures  for  acceptance  of  ships  from 
contractors  and  then  into  service.  However,  only  the  USN  states  the  purpose  of  ship 
acceptance  which  is  “to  ensure  delivery  to  the  Fleet  of  complete  ships,  free  from 
both  contractor  and  government  responsible  deficiencies”  [Ref.  35:p.  1]. 

Each  Navy  uses  similar  terminology,  but  these  have  different  meanings  which 
can  create  confusion.  A  comparison  of  terms  applicable  to  ship  acceptance  is  given 
in  Table  17. 

1.  Ship  Acceptance  Policy 

Ship  acceptance  and  Fleet  Introduction  within  the  USN  has  traditionally 
been  based  only  on  the  successful  completion  of  PAT&E  and  material  inspections. 
However,  with  the  treatment  of  ships  as  complete  systems,  as  for  the  DDG-51,  Fleet 
Introduction  is  now  based  also  on  OT&E  results. 

The  RAN  has  also  placed  emphasis  on  PAT&E  leading  to  Acceptance  into 
Naval  Service  (AINS),  although  OT-3  has  been  included  in  the  trials  program 
leading  to  AINS.  However,  these  trials  conducted  as  OT-3,  are  really  PAT&E  / 
DT&E  with  minimal  true  OT&E. 
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TABLE  17:  COMPARISON  OF  ACCEPTANCE  TERMINOLOGY 


USN  Term 

USN  Definition 

RAN  Term 

RAN  Definition 

Acceptance 

The  legal  act  of  accepting 
custody  of  a  new  construction 
ship  by  the  Navy  upon  delivery 
by  a  private  shipbuilder. 

Delivery 

When  the  ship  is  contractually 
delivered  from  the 

contractor’s  ownership  to  the 
Government. 

Delivery 

The  actual  assumption  of 
custody  by  the  Navy  incident 
to  acceptance.  The  date  of 
delivery  from  a  private 
shipyard  is  also  the  date  of 
acceptance. 

Preliminary 

Acceptance 

(PA) 

Determination  that  the  ship  has 
reached  both  material  and 
support  status  sufficient  to 
safely  and  effectively  proceed 
with  Navy  trials.  It  is  usually 
concurrent  with  Delivery. 

Fleet 

Introduction 

Approval  by  SECNAV  that  a 
ship  class  meets  the 
operational  effectiveness, 
operational  suitability,  safety 
and  material  standards  for 
service  use. 

Acceptance 
into  Naval 
Service 
(AINS) 

Acceptance  of  the  ship  by  CNS 
as  a  fully  operational  unit  of 
the  Heet,  based  on  operational 
performance  and  support 
levels  obtained  at  that  time. 

Operational 

Acceptance 

(OA) 

Carried  out  only  of  first  of 
class  to  supplement  the  AINS 
assessment  using  additional 
operational  and  support 
experience  with  the  Fleet  to 
refine  the  vessel  and  Naval 
Requirements. 

2.  Ship  Acceptance  Procedure 

The  procedure  for  USN  ship  acceptance  and  introduction  to  the  Fleet  as 
used  for  the  DDG-51  program  is  outlined  in  Figure  23.  A  noteworthy  aspect  of  the 
USN  schedule  is  the  exclusive  OT-3  period,  the  results  of  which  support  a  Fleet 
Introduction  decision. 


154 


ACTIVITIES 


Bnildersl 

TMals 


Hsiubd Uc«»ctUsk.t;^  w. 

n  Tnab _ &  Trials  - T  Avalabiitv 


MILESTONES 


AvaiabiMty 


OT-3 


Ac  cqitanoe 
/D«vei7 


Comnrission 


Reet 

In  traduction 


Figure  23:  USN  Ship  Acceptance  Schedule  and  Milestones 


The  Board  of  Inspection  and  Survey  (INSURV)  conducts  Acceptance  T rials 
and  Final  Contract  Trials  for  each  individual  new  ship  to  provide  independent 
verification  of  readiness  of  ships  for  acceptance  in  the  USN.  Based  upon  its  findings, 
the  Board  recommends  acceptance  or  final  settlement  of  the  contract.  The  Board 
identifies  material  conditions  which  represent  departures  from  the  USN  General 
Specifications  and  deficiencies  that  substantially  reduce  the  ship’s  fitness  for  naval 
service  or  degrade  its  ability  to  perform  its  primary  mission.[Ref.  34:p.  1]  On 
successful  trials  completion,  NAVSEA  accepts  the  ship  from  the  contractor. 

For  comparison,  the  RAN  ship  acceptance  schedule,  as  used  for  the 
Australian  Frigates  and  proposed  for  the  ANZAC  frigates,  is  outlined  in  Figure  24. 
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Figure  24:  RAN  Ship  Acceptance  Schedule  and  Milestones 


Of  note  in  this  schedule  is  the  combined  PAT&E  /  OT&E  period  supporting 
an  AINS  decision,  and  the  OT-4  phase  leading  to  OA.  OT-4  is  not  an  exclusive  T&E 
phase,  but  is  integrated  with  the  normal  fleet  operations  of  the  ship.  AINS  is  the  most 
significant  post  delivery  milestone  as  it  represents  acceptance  of  the  ship  into  the 
RAN  and  that  the  project  has  met  it’s  responsibilities. 

In  the  RAN,  Delivery  (contractual  acceptance)  of  the  ship  to  the 
Government  is  a  contractual  matter  and  is  managed  by  the  Project  Director.  CTE’s 
responsibility  is  to  conduct  the  Acceptance  Trials  and  report  the  results.  An 
Acceptance  Board  (AB)  is  established  to  provide  an  assessment  of  the  ship  and 
submit  recommendations  upon  which  CNS  can  base  his  acceptance  decision, 
however  the  AB  is  quite  different  from  the  INSURV  Board.  The  AB  is  usually 
established  for  each  new  ship  type  and  only  for  first  of  class.  The  President,  Vice- 
President  and  Board  members  serve  on  the  Board  part  time.  The  AB’s  sole  function 


156 


is  to  advise  CNS  on  acceptance  matters.  This  is  achieved  by  assessing  the  adequacy 
of  the  trials  plans,  examining  the  inspection  and  trials  reports,  attending  trials  where 
necessary  and  then  providing  an  overall  assessment  of  the  adequacy  of  the  ship 
against  the  user  requirement  The  Acceptance  Board  does  not  conduct  tests,  trials  or 
inspections,  and  is  essentially  a  board  of  review. 

Acceptance  into  Naval  Service  is  essentially  the  transfer/acceptance  of  the 
ship  from  the  acquisition  part  of  the  Navy  (ACMAT-N)  to  the  inservice  operations 
and  support  areas,  as  having  met  the  user  requirement  Post  delivery  trials  leading  to 
AINS  includes  the  SQT  period,  with  harbour  and  sea  phases. 

For  the  first  of  class,  it  will  normally  not  be  possible  to  trial  and  evaluate  all 
aspects  of  the  operational  performance  and  maintainability  prior  to  AINS.  This  OT- 
4  period  is  conducted  following  AINS  and  is  compiled  by  RANTAU  in  consultation 
with  the  Naval  and  Maritime  Headquarters  Staff.  It  comprises  a  series  of  trials  and 
evaluations  to  assess  operational  effectiveness  and  operational  suitability.  [Ref. 
41  :p.  15.14]  Typically,  the  Follow-on  T&E  program  will  last  from  12-24  months, 
depending  on  the  class  of  ship.  At  the  end  of  this  period  RANTAU  compiles  a  report 
in  consultation  with  Maritime  Headquarters  Staff.  The  AB  assesses  the  results  and 
provides  the  overview  to  ensure  the  program  is  comprehensive.  This  assessment 
serves  to  confirm  or  modify  the  assessment  at  AINS. 

3.  Ship  Acceptance  Summary 

Both  Navies  now  include  OT&E  in  their  post-delivery  trials  of  the  first  of 
class  to  ensure  that  the  ship  is  assessed  operationally  against  the  User  Requirement. 
However  the  RAN  OT-3  period  is  not  true  OT&E,  but  consists  mostly  of  DT&E  and 
PAT&E  with  some  OT&E  elements.  The  USN  INSURV  Board  conducts  acceptance 
trials  on  every  ship  but  looks  at  materiel  items  only,  which  is  different  to  the  RAN 
AB,  which  is  a  board  of  specialists  who  review  the  trials  and  results  of  the  first  of 
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class.  OT-3  within  the  RAN  needs  to  be  separated  from  the  DT&E  /PAT«feE  post 
delivery  events,  and  should  consist  of  true  OT&E.  This  would  enable  the 
Acceptance  Board  to  use  its  knowledge  and  experience  in  assessing  the  results  of 
OT«feE  against  the  user  requirement,  instead  of  assuming  a  quality  control  function 
over  what  is  essentially  PAT«feE  testing. 

G.  OT&E  PROCEDURES 

This  section  discusses  the  procedures  adopted  by  the  two  Navies  in 
implementing  OT&E  policy. 

1.  User  Requirement  to  OT&E  Test  Plan 

To  this  concruence  between  OT&E  test  documentation  and  the  user 
requirement,  both  Navies  generate  a  series  of  documents  through  the  acquisition 
process.  Although  the  documents  between  Navies  have  different  titles,  they  have 
similar  functions.  A  comparison  of  the  progression  of  documentation  leading  from 
the  user  requirement  to  the  OT&E  test  plans  is  given  in  Table  18. 

Both  Navies  can  trace  a  requirement  from  the  capability  need  through  the 
refining  requirements  documentation  to  specific  a  objective  in  the  TEMP,  to  the  test 
plan  and  ultimately  to  the  final  OT&E  report 
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TABLE  18:  COMPARISON  OF  REQUIREMENTS  DOCUMENT ATON 


USN 

Document 

RAN 

Document 

Function 

Document 

Mission  Need  Statement 

Capability 

Defence  Force 

(MNS) 

Need 

Capability  Proposal  (DFCP) 

Operational  Requirements 

Refined 

Required  Operational 

Document  (ORD) 

Requirement 

Characteristics  (ROC) 

Top  Level  Requirement 

Requirement 

Agreed  Ships 

(TLR) 

Specifics 

f'haracteristics  (ASC) 

Test  and  Evaluation 

Overall 

Test  and  Evaluation 

Master  Plan  (TEMP) 

T&E  Plan 

Master  Plan  (TEMP) 

OT&E 

OT&E 

Acceptance  Management  Plan 

Test  Plan 

Plan 

(AMP) 

2.  Assessment  of  Operational  Effectiveness 

The  RAN  has  found  it  difficult  to  define  exactly  what  is  required  to  assess 
operational  effectiveness.  The  RANTAU  guidance  on  the  conduct  of  OT&E  [Ref. 
53]  recognises  that  the  user  requirement  is  the  bench  mark  for  determining  the 
degree  to  which  a  product  is  effective.  Also  that  operational  effectiveness  is  best 
assessed  by  a  performance  demonstration  by  normal  operating  personnel  in  the 
normal  or  given  environment.  But  otherwise  provides  very  little  guidance  on  the 
assessment  of  operational  effectiveness.  Past  techniques  employed  involve 
analyzing  each  COI  and  employing  modified  Ship  Qualification  Trial  and  Fleet 
Exercise  Program  techniques  to  evaluate  them.  Use  has  also  been  made  of 
simulation,  weapon  analysis  and  exercise  analysis.  The  lack  of  suitable  range 
facilities  has  also  limited  the  assessment  of  operational  effectiveness. 
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The  USN,  by  comparison,  has  a  more  effective  method  of  assessing 
operational  effectiveness.  As  do  the  RAN,  the  USN  examines  each  COI  and  related 
operational  effectiveness  objective,  and  decides  what  needs  to  be  known  to  enable 
each  objective  to  be  assessed  [Ref.  15:p.  9-1].  They  ensure  that  the  appropriate 
environments,  threats,  etc.  are  included  and  that  sufficient  data  will  be  generated  to 
address  the  COI  and  objectives.  They  focus  on  achieving  statistical  relevance  where 
possible,  and  making  the  tests  as  objective  as  possible.  DOT&E  works  closely  with 
the  USN  to  help  determine  how  many  test  assets  are  required  to  achieve  a  certain 
degree  of  confidence  that  the  results  are  correct.  A  test  may  need  to  be  repeated  a 
number  of  times  to  gain  statistical  relevance.  However,  if  this  is  not  possible,  or  the 
direct  proving  of  an  ability  is  not  possible  (as  in  the  surface  warfare  example),  then 
the  test  needs  to  be  designed  so  that  what  is  observed  in  the  test  program  is  projected 
to  be  what  would  be  observed  in  reality. [Ref.  19][Ref.  21]  These  situations  require 
expertise  and  Judgement,  and  compromise  between  the  authorities  involved,  to 
make  the  tests  as  objective  as  possible. 

3.  Assessment  of  Operational  Suitability 

The  principles  of  operational  suitability  are  common  between  the  USN  and 
RAN,  and  similar  aspects  are  assessed.  Within  the  USN,  COMOPTEVFOR  reviews 
the  Integrated  Logistic  Support  Plan  (ILSP)  against  the  user  requirement.  Following 
this  review,  COMOPTEVFOR  determines  the  suitability  tests  required  based  on 
expected  reliability,  degree  of  confidence,  thresholds,  etc.  required  to  be  achieved. 
These  determine  the  scope  and  length  of  the  assessment.  Reliability,  maintainability 
and  availability  (RM&A)  data  is  collected  and  sent  to  the  normal  inservice  analyst 
of  RM&A  data,  the  Naval  Weapons  Analysis  Centre  (NWAC)  for  analysis. 
Specialist  analysts  within  OPTEVFOR  design  the  tests  and  determine  measures  of 
suitability.  They  also  evaluate  and  analyse  the  adequacy  of  logistic  supportability. 
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However,  OPTEVFOR  also  relies  on  operational  personnel  to  use  their  experience 
and  knowledge  of  the  system  to  identify  inadequate  logistic  support. 

The  assessment  of  operational  suitability  in  the  RAN  has  been 
controversial,  particularly  the  assessment  of  supportability.  Before  1987,  the 
assessment  of  operational  suitability,  and  ILS  in  particular,  was  rather  subjective  and 
often  controversial.  A  new  approach  was  taken  in  1987  and  has  continued  to  be 
refined.  The  assessment  of  ILS  is  now  conducted  in  two  phases: 

•  ILS  Structural  Review 

•  ILS  Operational  Review 

The  ILS  Plan  is  reviewed  and  discussions  with  the  functional  areas 
handling  the  support.conducted  prior  to  operational  testing.  During  the  operational 
testing  period,  the  ship’s  ILS  related  documentation  and  records  are  reviewed, 
maintainability  demonstrations  are  performed  and  a  supportability  assessment  is 
conducted.  The  RAN  is  currently  developing  a  routine  in-service  RM&A  data 
collection  and  analysis  system,  however,  until  that  is  operational,  the  data  is 
collected  and  analysed  on  an  ’as  required’  basis  by  the  Trials  Authority. 

In  summary,  the  principles  of  operational  suitability  assessment  are 
common  between  the  USN  and  RAN.  Although  assessments  by  the  RAN  have 
become  less  objective  and  more  subjective  over  recent  years,  the  assessment  of 
RM&A  is  still  in  its  infancy.  Both  Navies  recognise  the  importance  of  data 
collection,  however,  the  RAN  needs  experienced  personnel  to  design  tests  and 
analyse  results. 

4.  OT&E  Coordination 

The  USN  recognises  the  importance  of  coordination  between  authorities  in 
the  successful  achievement  of  T&E.  The  USN  T&E  coordinator  is  responsible  to  the 
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Director,  Test,  Evaluation  and  Technology  Requirements  (N091).  They  are 
essentially  the  OPNAV  Staff  coordinator  for  Navy  T&E,  providing  a  primary 
contact  point  for  all  parties  and  setting  up  T&E  briefings  and  meetings.  Their 
responsibilities  include  the  chairmanship  of  the  Test  and  Evaluation  Coordination 
Group  (TECG)  for  each  major  program. (Ref.  37]  TECGs  are  used  for  complex, 
multifaceted  programs  which  require  extensive  T&E  coordination.  Membership  of 
a  TECG  includes  the  Program  Manager,  the  Sponsor,  COMOPTEVFOR,  a  logistics 
coordinator  and  others  as  appropriate  (such  as  a  PREINSURV  representative). 
TECG  recommendations  are  considered  for  inclusion  in  the  TEMP.  [Ref.  1  l;p.  15] 
Some  of  the  functions  of  a  TECG  are  the  early  definition  of  terms,  measures  of 
effectiveness  and  how  these  are  to  be  measured,  and  the  criteria  for  acceptable  or  not 
acceptable.  Of  note  is  the  formation  of  a  TECG  does  not  imply  a  joint  test  team 
approach.  Each  T&E  agency  remains  fully  and  solely  responsible  for  conducting 
and  reporting  the  types  and  phases  of  T&E  for  which  it  is  accountable. 

In  the  RAN,  a  T&E  Planning  Group  (TEPG)  is  formed  prior  to  TEMP 
development  under  the  leadership  of  the  T&E  manager  for  the  project.  Foundation 
members  of  the  TEPG  are  the  Project  Director,  Project  Sponsor,  the  Design 
Approval  Authority  and  RANTAU,  with  other  parties  seconded/called  upon  as 
required.  [Ref.  44:p.  5]  The  TEPG  analyses  T&E  requirements  and  estimates 
resources  required  to  meet  the  requirements. 

The  function  of  these  groups  is  similar,  however,  the  USN  TECG  is  led  by 
the  Director,  Test,  Evaluation  and  Technology  Requirements  (N091)  under  the 
CNO,  whereas  the  RAN  TEPG  is  led  by  each  individual  Project  Director  through 
this  T&E  manager.  Despite  their  best  intentions.  Project  Directors  are  essentially 
driven  by  cost  and  schedule  considerations,  not  the  overall  T&E  adequacy  of  their 
project.  To  be  truly  objective,  each  TEPG  should  be  chaired  by  the  authority 
responsible  for  ensuring  the  adequacy  of  the  RAN’s  overall  T&E  program. 
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5.  OT&E  Reporting 

Reports  provide  the  OT&E  authority’s  conclusions  regarding  a  system’s 
operational  effectiveness  and  suitability,  and  his  recommendations  regarding  the 
systems  future,  i.e.,  acceptance,  further  development,  additional  OT&E,  etc.  The 
USN  system  has  three  basic  principles  for  OT&E  reports: 

•  Impartial 

•  Complete  and  Thorough 

•  Sole  Reporter 

The  conduct  of  OT&E  must  be  impartial  and  not  influenced  by  the  program 
manager  or  contractors  during  testing  and  analysis.  COMOPTEVFOR’s 
conclusions  are  based  on  a  complete  and  thorough  analysis.  A  report  covers  a 
complete  OT&E  phase  (e.g.,  OT-IIIA),  relating  the  test  results  to  the  COIs  and 
addressing  the  objectives  stated  in  the  TEMP.  Quick-look  and  interim  reports  are 
usually  sent  only  if  the  testing  could  not  be  completed  or  when  directed  by  the  CNO. 
Although  the  final  report  may  be  a  surprise  to  other  authorities.  OT&E  results  and 
analysis  are  reported  through  one  authority  only.  Comments  from  the  Commanding 
Officer  of  the  ship  under  test  ship,  for  example,  are  sent  only  to  COMOPTEVFOR. 
All  operational  test  data  is  considered  to  be  the  owned  by  COMOPTEVFOR  until 
the  final  report  is  signed. 

Within  the  RAN,  the  OT&E  Authority  coordinates  the  issue  of  trials 
reports,  however,  they  are  issued  on  completion  of  each  segment  of  OT&E  e.g.,  o/c 
ASW  phase  of  OT-3  etc.  Quicklook  reports  are  also  routinely  sent  on  completion  of 
each  week’s  testing.  The  ship  under  test  also  often  provides  a  report,  however,  it  is 
usually  forwarded  to  their  operational  authority  and  the  Project  in  addition  to  the 
Trials  Authority.  For  the  results  of  PAT&E,  rapid  feedback  is  necessary  to  ensure 
timely  rectification  of  defects.  However,  OT&E  results  may  indicate  system 
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deficiencies  against  user  requirements  rather  than  equipment  defects,  so  the  results 
need  to  be  fully  analysed  and  the  implications  assessed  before  being  reported  to  a 
wider  audience.  The  RAN  practice  of  OT&E  reporting  probably  stems  again  from 
the  lack  of  delineation  between  types  of  T&E  and  the  combined  PAT&E  /  OT&E 
nature  of  the  RAN’s  OT-3  testing. 

6.  OT&E  Personnel  Selection  and  Training 

OT&E  is  a  specialised  discipline,  with  its  own  philosophy  and 
methodology,  and  so  requires  a  specialist  approach  with  knowledge  and  experience 
to  make  it  effective. 

a.  Selection 

The  U.S.  community  involved  with  ship  OT&E  select  and  conduct 
limited  training  for  their  military  personnel.  Although  there  are  no  special  selection 
requirements  for  detailing  military  personnel  for  service  as  Test  Directors  in 
OPTEVFOR,  broad  ship  operations  experience  and  combat  system  knowledge  are 
preferred.  For  particular  programs,  OPTEVFOR  make  use  of  subject  area  experts, 
either  resident  or  borrowed  from  a  non-interested  party.  Military  personnel  required 
by  DOT&E  are  selected  by  background  and  expertise,  preferably  war  college 
graduates  with  joint  experience. 

No  special  selection  criteria  is  involved  for  posting  OT&E  personnel 
within  the  RAN,  although  like  the  USN,  broad  ship  operations  experience  and 
combat  system  knowledge  are  preferred.  Very  few  personnel  have  had  any  T&E 
experience  prior  to  joining  the  OT&E  authority.  Also  currently  the  OT&E  authority 
is  viewed  by  some  as  not  being  career  enhancing  for  warfare  personnel. 


b.  Training 

COMOPTEVFOR  conducts  a  four  day  Operational  Test  Director’s 
(OTD)  overview  course  covering  the  major  areas  of  OT&E.  Further  details  of  this 
course  are  included  in  Appendix  A.  They  also  run  adhoc  segment  courses  which 
provide  acquaints  or  updates  on  OT&E  subjects  e.g.,  analysis,  test  plan  development 
and  threat  updates.  To  assist  their  personnel  in  managing  OT&E,  OPTEVFOR 
publishes  the  OT&E  Director’s  Guide  which  documents  their  philosophy  and 
methodology.  [Ref .  21]  Other  authorities  involved  with  OT&E  also  send  personnel 
to  attend  this  course.  Although  DOT&E  does  not  conduct  courses  for  new  staff,  a 
one  month  handover  is  usually  required. 

The  RAN  has  no  training  courses  on  OT&E,  or  on  T&E  in  general.  A 
number  of  officers  have  completed  the  USN  Operational  Test  Directors  Course  in 
recent  years  which  has  improved  the  knowledge  and  understanding  of  OT&E  within 
CTE’s  organisation  considerably.  These  courses  have  been  arranged  on  an  ad-hoc 
basis  through  overseas  visit  submissions  rather  than  as  pre-requisite  courses  for 
particular  billets.  For  guidance  of  trials  personnel  and  other  authorities,  CTE  has 
developed  a  number  of  Standard  Operating  Procedures  addressing  certain  aspects  of 
OT&E. 

c.  Career  Path 

Although  some  personnel  may  develop  into  OT&E  specialists,  no 
career  path  in  OT&E  is  consciously  provided  in  the  USN,  and  no  naval  personnel 
currently  serving  in  DOT&E  have  served  previously  in  OPTEVFOR.  Similarly  in 
the  RAN,  no  career  path  in  T&E  is  provided. 

To  be  effective,  OT&E  ideally  requires  current,  experienced, 
knowledgable  personnel  with  high  professional  credibility  within  their  field  of 
expertise.  To  achieve  this,  selection  of  suitable  personnel,  adequate  training  and 
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good  guidance  documentation  is  required,  and  preferably  a  career  path  where  OT&E 
is  seen  as  suitable  career  progression  by  such  capable  personnel. 

H.  OT&E  IMPLEMENTATION 

Recent  ship  projects  illustrate  how  these  policies  and  procedures  are 
implemented.  The  USN  DDG-51  program  employs  a  new  hull  with  a  new, 
evolutionary  combat  system.  Noteworthy  aspects  of  DDG-51  OT&E  included  OT- 
II  on  both  combat  and  propulsion  systems  LBTS  to  gain  an  early  indication  of 
potential  operational  effectiveness  and  suitability  which  supported  the  decision  to 
approve  the  DDG-51  to  go  beyond  Low  Rate  Initial  Production.  Another 
characteristic  of  the  DDG-51  program  was  “Whole  Ship”  OT&E  where  the  whole 
ship  is  viewed  as  an  integrated  warfare  system,  rather  than  as  a  platform  for 
individual  warfare  systems.  OT-3  was  conducted  in  a  dedicated  period  following 
CSSQT  and  Final  Contract  Trials  (but  before  the  workup)  to  support  a  Fleet 
Introduction  decision.  It  consisted  of  fully  assessed,  free  play  multi-threat  scenarios 
with  the  emphasis  on  operational  realism. 

The  implementation  of  OT&E  policy  within  the  RAN  is  best  demonstrated  by 
reviewing  examples  of  ship  OT&E  conducted  to  date,  and  that  planned  for  ships  due 
to  enter  service  in  the  near  future. 

The  three  RAN  DDGs  underwent  a  major  modernisation  and  reht  over  the 
period  1987-1991.  The  major  modernisation  occurred  with  the  ships’  combat 
system,  command  and  control  system,  sensors  and  weapons,  with  the  aim  of 
increasing  the  ‘supportability’  of  the  systems  rather  than  increasing  the  ships’ 
capability.  As  a  low  operational  risk  project,  the  OT&E  followed  the  established 
RAN  policy,  however,  it  was  the  first  to  actually  implement  an  OT-4  period  to  more 
fully  assess  operational  effectiveness  and  operational  suitability  leading  to  OA. 
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Two  US  FFG-7  class  frigates,  known  as  Australian  Frigates  (AF),  are  being 
built  in  Australia.  The  AF  is  a  low  operational  risk  project,  except  for  the  addition 
of  the  Mulloka  sonar.  The  OT&E  followed  the  established  RAN  policy,  however,  it 
failed  to  plan  an  OT-4  period  to  more  fully  assess  operational  effectiveness  and 
operational  suitability  of  the  Mulloka  sonar. 

Australia  and  New  Zealand  are  cooperating  in  the  building  of  the  ANZAC 
frigate.  Essentially  a  MEKO  200  design  incorporating  a  combination  of  European 
and  US  equipment,  the  design  is  classified  as  low  risk,  although  modifications  to  the 
design  were  necessary  to  meet  RAN  requirements.  Although  only  in  the  early 
stages,  the  planned  OT&E  for  the  ANZAC  ship  follows  the  established  RAN  policy, 
with  the  OT-3  period  combined  with  the  SQT,  being  conducted  prior  to  AINS.  The 
ship  class  has  a  higher  operational  risk  than  either  the  modernised  DDG  or  the  AF. 

The  RAN  is  building  six  boats,  known  as  the  Collins  class,  to  the  Swedish 
Kockums  Type  471  design.  They  will  be  the  first  submarines  to  be  constructed  in 
Australia  and  will  have  a  unique  equipment  fit  These  submarines  possibly  have  the 
highest  operational  risk  of  all  current  major  RAN  combatant  projects.  Consequently 
the  increased  need  for  OT&E  has  been  recognised.  The  planned  OT&E  program 
demonstrates  a  departure  from  the  usual  RAN  OT&E  policy,  and  reflects  a  growing 
awareness  of  the  need  to  separate  DT  and  OT  activities,  and  to  include  a  dedicated 
OT-3  period  after  the  usual  SQT  and  ORE. 

These  examples  of  RAN  implementation  of  ship  OT&E  policy  illustrates  a 
number  of  key  issues.  The  operational  risk  of  ship  projects  within  the  RAN  is 
increasing,  leading  to  a  growing  awareness  for  the  increased  need  for  ship  OT&E. 

Significant  differences  in  OT&E  implementation  between  the  USN  and  RAN 
are,  first,  that  the  RAN  does  not  generally  conduct  OT&E  to  support  the  production 
or  purchase  decision  of  ships.  Second,  the  RAN  has  relied  on  the  results  of  a 
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combined  SQT  and  OT-3  period  to  provide  a  basis  for  AINS,  whereas  the  USN 
conduct  a  “whole  ship”  dedicated  OT-3  phase  to  support  Fleet  Introduction. 

L  SUMMARY 

The  USN  and  RAN  have  similar  acquisition  systems  and  similar  fundamental 
definitions  of  T&E.  However,  the  OT&E  systems  differ  in  a  number  of  significant 
areas.  Both  Navies  appreciate  the  importance  of  T&E  in  the  acquisition  process, 
however,  the  level  of  importance  of  T&E,  and  OT&E  in  particular,  differs  greatly. 
The  USN  recognises  that  without  OT&E,  it  risks  deploying  systems  which  cannot 
adequately  perform  significant  portions  of  their  missions,  thus  degrading  its 
deterrent/defensive  capabilities  and  endangering  the  safety  of  military  personnel 
who  operate  and  maintain  the  systems.  The  importance  of  OT&E  within  the  RAN  is 
not  specifically  addressed  in  any  T&E  policy  document.  The  USN  places  emphasis 
on  OT&E  leading  to  full  scale  production  where  the  majority  of  a  project’s  funding 
is  spent.  In  contrast,  the  RAN  system  places  a  high  priority  on  the  post  production 
phase. 

The  RAN  has  adapted  the  more  detailed  USN  T&E  definitions  to  suit  its  own 
requirements.  During  this  process,  the  RAN  has  lost  the  distinction  between  the 
types  (rf  T&E  and  often  confuses  OT&E  with  PAT&E  /  DT&E.  USN  OT&E  policy 
has  its  basis  in  statutes  approved  by  Congress.  The  RAN  has  no  such  formal  basis 
and  its  OT&E  policy  is  embedded  in  the  more  general  T&E  policy. 

The  USN  OT&E  organisation  is  well  structured  with  comprehensive 
responsibilities  for  each  participant,  and  clearly  delineated  from  the  DT&E 
structure.  Following  a  number  of  recent  reorganisations,  the  RAN’s  OT&E  structure 
is  in  disarray,  with  no  clear  pdicy  maker  for  T&E  in  general,  or  OT&E  in  particular, 
lacks  functional  authorities  with  T&E  knowledge  to  advise  Project  Directors,  and  its 
OT&E  authority  is  buried  in  the  in-service  area  of  the  Maritime  Command. 
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Both  Navies  have  similar  policies  and  procedures  for  acceptance  of  ships  from 
contractors  and  then  into  service.  Each  Navy  uses  similar  terminology,  but  these 
have  different  meanings.  OT&E  results  in  the  USN  is  now  having  a  greater  input  on 
ship  acceptance.  Although  the  RAN  has  traditionally  included  OT&E  in  support  of 
AINS,  the  nature  of  the  testing  has  really  been  more  DT&E  /  PAT&E  rather  than  true 
OT&E. 

The  USN  OT&E  system  has,  by  necessity,  developed  into  a  well  o.-'ganised, 
well  documented  and  effective  system.  The  RAN  system,  on  the  other  hand,  suffers 
from  confusing  T&E  policy  documentation,  a  weak  OT&E  organisation  structure 
and  a  general  lack  of  OT&E  knowledge  and  appreciation.  The  determination  of  the 
OT&E  required  has  traditionally  not  been  achieved  early  enough  in  the  life  of  a 
project,  and  so  project  funding  has  not  included  the  provision  for  OT&E.  However, 
OT&E  awareness  and  knowledge  within  the  RAN  is  growing  as  evidenced  by  the 
proposed  Collins  class  post  delivery  trials. 

The  RAN  OT&E  system,  although  originally  based  on  USN  OT&E  philosophy, 
now  differs  in  the  importance,  interpretation,  application  and  focus  of  OT&E  As  a 
result  of  organisation  changes,  it  now  lacks  the  ability  to  effectively  manage  overall 
T&E,  let  alone  OT&E,  within  the  RAN.  To  achieve  efficient  and  effective  trials  and 
acceptance  of  the  new  higher  risk  combatants  currently  under  construction,  the  RAN 
OT&E  system  needs  to  be  revised  to  fully  support  the  acquisition  process. 
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VI.  A  MODEL  FOR  OT&E  IN  THE  RAN 


This  chapter  proposes  a  model  for  the  OT&E  system  in  the  RAN.  The  OT&E 
principles  derived  from  the  USN  system  are  used  to  develop  a  system  to  suit  RAN 
requirements. 


A.  TEST  &  EVALUATION  DEHNITIONS 

The  basis  for  the  OT&E  system  are  clear  definitions  of  the  types  and  phases  of 
T&E.  The  RAN  should  continue  to  recognise  the  three  types  of  T&E  associated  with 
ships.  The  current  definitions  listed  in  the  CPO  Manual  [Ref.  38]  should  be  amended 
to  better  suit  the  Australian  Defence  environment. 

1.  Development  Test  and  Evaluation  (DT&E) 

The  DT&E  definition  remains  unchanged  as  follows: 

DT&E  is  conducted  to  assist  the  engineering  design  and  development  process, 
and  to  verify  attainment  of  technical  performance  specifications  and 
objectives. 

The  proposed  phases  of  DT&E  are  detailed  in  Table  19.  The  three  phases 


TABLE  19:  PROPOSED  PHASES  OF  DT&E 


Current  RAN  Description 

DT 

Proposed  RAN  Description 

Validation  of  design  concept. 

DT-1 

Validation  of  design  concept. 

Proving  design. 

DT-2 

Demonstration  that  design  meets  specifications. 

Demonstration  that  production 
meets  required  technical 
characteristics. 

DT-3 

Demonstration  that  production  meets  required 
technical  characteristics  or  establish  standards  for 
first  of  class.  PAT&E  is  a  form  of  DT-3. 
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are  retained,  however,  the  DT-2  definition  is  amended  to  clarify  the  meaning  of 
“proving  design”,  and  the  DT-3  phase  now  recognises  that  PAT&E  is  a  form  of 
DT&E. 

2.  Production  Acceptance  Test  and  Evaluation  (PAT&E) 

The  definition  of  PAT&E  is  revised  to  recognise  that  it  is  conducted  not 

only  during  the  contract  period,  but  also  during  subsequent  RAN  testing: 

PAT&E  is  conducted  on  production  items  to  ensure  systems  meet  technical 
specifications  and  requirements,  and  is  a  type  of  DT&E. 

The  proposed  phases  of  PAT&E  are  listed  in  Table  20.  Phases  PAT-0  to  PAT- 
5  remain  unchanged.  PAT-6  is  amended  to  delete  any  connection  with  01-3,  as  under 
the  proposed  model  OT-3  will  now  be  conducted  separately.  The  PAT-7  phase  is 
deleted  as  no  PAT&E  is  conducted  during  the  OT-4  phase  of  testing. 


TABLE  20:  PROPOSED  PHASES  OF  PAT&E 


Current  RAN  Description 

PAT&E 

Proposed  RAN  Description 

Design  and  engineering 
development  tests 

PAT-0 

Design  and  engineering 
development  tests 

Production  and  bum-in  tests 

PAT-1 

Production  and  bum-in  tests 

Environmental  qualification  tests 

PAT-2 

Environmental  qualification  tests 

System  development  tests 

PAT^ 

System  development  tests 

Harbour  testing 

PAT-4 

Harbour  testing 

Sea  testing 

PAT-5 

Sea  testing 

Operational  and  Qualification  Trials 
(conducted  with  OT-3) 

PAT^ 

Certification  and  Qualification 

Trials 

Follow  on  Operational  T&E 
(conducted  with  OT-4) 

PAT-7 

Nil. 
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3.  Operational  Test  and  Evaluation  (OT&E) 

The  definition  of  OT&E  is  revised  along  USN  lines,  so  that  it  more 

accurately  describes  the  purpose  and  nature  of  OT&E: 

OT&E  is  conducted  to  determine  a  system 's  operational  effectiveness  and 
operational  suitability,  identify  system  deficiencies  and  the  need  for  potential 
modifications  to  meet  established  OT  thresholds,  and  develop  tactics.  OT&E 
has  three  distinguishing  characteristics: 

•  It  is  conducted  in  an  operationally  representative  environment. 

•  It  is  conducted  on  production  representative  equipment  using  fleet  personnel 
for  operation  and  maintenance. 

•  It  is  conducted  against  a  threat-representative  simulated  enemy  carrying  out 
threat  tactics  per  the  latest  threat  assessment. 

The  proposed  phases  of  OT&E  are  described  in  Table  21.  These  amended 
definitions  include  reference  to  the  acquisition  milestones  they  support.  The 
reference  to  OPEVAL  in  the  OT-2  phase  has  been  deleted,  as  the  RAN  uses  the  term 
OPEVAL  as  an  abbreviation  for  an  operational  evaluation  in  any  OT  phase.  The 
reference  to  a  prototype  in  OT-2  has  also  been  deleted  as  OT-2  may  also  be 
conducted  in  a  LBTS,  or  on  a  sample  system  if  the  proposed  purchase  is  “off  the 
shelf’.  The  OT-3  phase  is  revised  to  delete  the  Certification  /Qualification  Trials  as 
they  are  now  recognised  as  PAT&E.  The  definition  of  OT&E  includes  the  use  of 
normal  fleet  personnel  so  this  wording  has  been  deleted  from  the  phase  definitions. 
The  OT-5  phase  is  amended  to  cover  ail  in-service  OT&E  which  could  be  conducted 
to  assess  new  applications,  new  tactics,  revised  threat,  etc.  Reference  to  defect 
rectification  is  deleted  also  as  it  is  a  sub-phase  of  OT-4,  and  the  assessments  of 
modernisations  and  modifications  are  deleted  as  they  would  be  new  OT-2  or  OT-3 
phases. 
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TABLE  21:  PROPOSED  PHASES  OF  OT&E 


Current  RAN  Description 

OT 

Proposed  RAN  Description 

Operational  assessment  of  the 
development  proposal 

OT-1 

Operational  assessment  of  the 
development  proposal  to  support  Full 
Scale  Engineering  Development 
approval. 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  suitability  of  a 
prototype  (OPEVAL)  to  support 
proceeding  to  full  production 

OT-2 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  suitability  to  support 
a  production  or  purchase  decision. 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  suitability  on 
production  of  ship/aircraft/system, 
normally  in  independent  operations 
using  normal  Fleet  personnel. 
Certiflcation/Qualiflcation  Trials 
include  limited  reliability, 
maintainability,  availability  and 
logistic  supportability  assessments. 

OT-3 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  operational 
suitability  on  production  of  ship/ 
aircraft/system  to  support  Acceptance 
into  Naval  Service.  Includes  limited 
reliability,  maintainability,  a\  ailability 
and  logistic  supportability 
assessments. 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  operational 
suitability  on  production  of  ship/ 
aircraft/ system  using  normal  Fleet 
personnel  in  a  multi-force,  multi-threat 
environment.  Includes  detailed 
reliability,  maintainability,  availability 
and  logistic  supportability 
assessments. 

OT-4 

Demonstration  of  achievement  of 
program  requirements  for  operational 
effectiveness  and  operational 
suitability  on  production  of  ship/ 
aircraft/system  in  a  multi-force,  multi¬ 
threat  environment  to  support 
Operational  Acceptance.  Includes 
detailed  reliability,  maintainability, 
availability  and  logistic  supportability 
assessments. 

Follow-on  OT  for  assessment  after 
modernisations,  new  applications  or 
defect  rectifications  after  OT-4. 

OT-5 

In-service  OT&E,  which  could  include 
new  applications,  new  tactics,  revised 
threat,  etc. 
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4.  T&E  Types  in  Total  Ship  Test  Program 

The  Total  Ship  Test  Program  (TSTP)  concept  is  now  well  established  in 
RAN  shipbuilding  projects,  although  modified  from  the  U.S.  system.  These  refined 
T&E  phases  can  be  incorporated  into  an  amended  TSTP  However,  instead  of  being 
grouped  together  as  in  the  current  TSTP,  the  DT&E  and  OT&E  events  are  separate 
categories  of  testing.  Because  these  separate  DT&E,  PAT&E  and  OT&E  events  are 
integrated  within  the  TSTP,  these  trials  are  termed  collectively  as  “Integrated  Tests 
and  Trials”,  which  is  an  extension  of  the  current  RAN  application  of  this  term.  The 
proposed  categories  of  ITT  are  listed  in  Table  22. 


TABLE  22:  CATEGORIES  OF  ITT 


nr  Part 

ITT  Category 

T&E  Type 

Description 

ITT  Part  1 

ITT-0* 

PAT^ 

Design  and  engineering  development  tests 

ITT-l 

PAT-l 

Production  and  bum-in  tests 

nT-2  * 

PAT-2 

Environmental  qualification  tests 

lTT-3* 

PAT-3 

System  development  tests 

nr^ 

PATA 

Harbour  testing 

rrT-5 

PAT-5 

Sea  testing 

ITT  Part  2 

nT-6 

DT-3/ PAT-6 

Certification  and  Qualification  Trials 

ITT-7* 

ar-3 

Operational  T&E 

ITT  Parts 

ITT-S* 

OT^ 

Follow  on  Operational  T&E 

Note:  *  Only  performed  on  first  of  class 


The  Certification  and  Qualification  trials  (nT-6)  are  classified  as  DT&E  for 
the  first  of  class  only,  as  standards  of  performance  are  established.  For  follow-on 
ships  these  trials  are  classed  as  PAT&E,  as  the  standards  established  by  the  first  of 


174 


class  become  the  benchmark  to  be  achieved  by  follow-on  ships.  ITT-0,  ITT-2,  ITT- 
3  and  ITT-7  tests  are  performed  only  on  the  first  of  class.  ITT-8  testing  would  only 
be  performed  once  for  a  ship  class,  but  may  be  conducted  progressively  on  any  ship 
of  the  class. 

R  T&E  IN  THE  ACQUISITION  PROCESS 

To  ensure  the  progressive  assessment  of  operational  effectiveness  and 
operational  suitability  during  the  acquisition  process,  OT&E  is  scheduled  towards 
the  end  of  each  acquisition  phase,  usually  after  the  DT  phase.  The  proposed  T&E 
schedule  is  shown  in  Figure  25.  No  change  is  suggested  to  the  current  relationship 
between  OT&E  phases  and  acquisition  milestones,  however,  the  figure  separates 
DT&E/PAT&E  from  OT&E  to  clearly  differentiate  these  different  types  of  testing. 


FULLSCALE 

PROGRAM 

ENGINEERING 

PRODUCTION 

PRELIM  aINS  OA 

INITUTION 

DEVELOPMENT 

APPROVAL 

APPROVAL 

ACCEPT 

PHASE  2 

PHASE  2 

DEMONSTRATION 

AND 

EVALUATION 

FULLSCALE 

ENGINEERING 

DEVELOPMENT 

DT-1 

DT-2 

DT&E/PAT&E 

OT&E 


Figure  25:  Proposed  Test  and  Evaluation  in  the  Acquisition  Process 
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As  the  construction  of  proven  overseas  ship  designs  adapted  to  RAN 
requirements,  does  not  require  the  conduct  of  OT-2  prior  to  a  production  or  purchase 
decision,  the  first  opportunity  to  evaluate  potential  operational  effectiveness  and 
operational  suitability  may  be  in  a  LBTS  after  the  production  decision  has  been 
made.  To  ensure  consistency  of  terminology,  this  testing  should  not  be  termed  OT- 
2  as  it  does  not  support  a  production  or  purchase  decision.  It  should  be  classified  as 
the  first  phase  of  OT-3. 

C.  RAN  OT&E  ORGANISATION 

The  objective  of  an  OT&E  organisation  structure  is  to  provide  effective  OT&E 
to  support  the  acquisition  process,  and  hence  the  combat  effectiveness  of  the  RAN. 
The  proposed  OT&E  structure  is  shown  in  Figure  26.  The  effectiveness  of  OT&E 


Figure  26:  Proposed  RAN  OT&E  Organisation 
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can  be  achieved  by  devoting  appropriate  resources  of  both  personnel  and  funding, 
by  considering  the  requirements  of  all  functional  areas  concerned  with  OT&E,  and 
by  coordinating  these  different  functions. 

1.  Policy  Maker  and  Overseer 

An  authority  is  required  to  formulate  OT&E  policy,  and  to  evaluate  and 
oversee  the  OT&E  process.  OT&E  policy  should  come  from  an  operational,  not 
technical,  functional  area.  HQADF  now  contains  the  centralised  force  development 
and  user  requirements  organisation.  As  OT&E  is  the  feedback  of  the  performance  of 
the  final  product  against  the  user  requirement,  then  ideally,  the  head  policy  maker 
and  overseer  of  OT&E  should  be  within  HQADF  or  DoD,  along  the  lines  of  the  U.S. 
system. 

A  shorter  term,  RAN-only,  solution  would  be  to  include  the  overseer  and 
policy  maker  function  within  the  revised  Director  General  Naval  Policy  and  Warfare 
(DGNPW)  organisation  under  DCNS.  Although  this  area  has  recently  been  reduced, 
DGNPW  still  retains  the  responsibility  for  naval  warfare  information  and  expertise. 
The  Director  of  Naval  Warfare  (DNW),  under  DGNPW,  provides  an  information 
brokerage  service  to  senior  level  management.  He  currently  has  a  secondary 
function  of  providing  naval  warfare  input  to  the  Defence  Capability  Proposals 
developed  by  HQADF,  and  providing  advice  on  naval  warfare  operations, 
equipment  and  personnel  aspects.  [Ref.  55:p.  A-4]  The  responsibility  of  providing 
advice  and  input  on  OT&E  matters,  as  part  of  the  OT&E  policy  maker  and  overseer 
function  for  the  RAN,  is  seen  as  a  natural  extension  to  these  current  responsibilities. 

2.  Sponsor 

The  Sponsor  of  major  naval  acquisitions  is  DGFD(Sea)  within  HQADF. 
Traditionally  in  the  RAN,  it  has  been  the  Sponsor’s  responsibility  to  initially 
determine  the  T&E  requirements  in  an  acquisition  program  and  to  detail  those 
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requirements  in  the  capability  proposal.  To  be  able  to  determine  the  appropriate 
nature  and  extent  of  OT&E  required,  the  Sponsor  needs  access  to  OT&E  knowledge 
and  experience.  A  close  relationship  needs  to  be  maintained  by  DGFD(Sea)  with  the 
OT&E  policy  maker  and  the  OT&E  authority  (Note  dotted  line). 

3.  Developing  Agency 

The  Materiel  Division  under  ACMAT-N,  performs  the  Developing  Agency 
function  within  the  RAN.  Currently  major  combatant  projects  establish  a  T&E 
manager,  with  the  larger  ones  also  establishing  a  manager  exclusively  for  post 
delivery  trials.  For  example,  the  Australian  Frigate  Project  has  a  Post  Delivery  Test 
and  Trials  manager,  and  the  New  Submarine  Project  has  just  established  a 
Submarine  Project  Transition  and  AINS  manager.  However,  ACMAT-N  does  not 
have  a  central  T&E  policy  authority  to  advise  these  T&E  managers.  Although  the 
Director  of  Naval  Equipment  Production  (DNEP)  has  responsibility  for  providing 
specialist  technical  advice  on  technical/engineering  activities,  which  includes  trials 
and  acceptance  of  equipment  [Ref.  64],  he  specialises  more  in  advice  for  contract 
management  than  for  post  delivery  T&E. 

To  assist  all  RAN  projects  with  T&E  matters,  an  authority  is  required 
within  the  Materiel  Division  whose  responsibilities  would  include  assisting  Project 
Directors  in  complying  with  policies,  incorporating  lessons  learned  from  previous 
projects  and  reviewing  the  T&E  aspects  of  projects  at  major  milestones.  Although 
primarily  a  DT&E  policy  authority,  this  office  would  also  have  an  input  into  the 
OT&E  process  by  reviewing  the  TEMP,  and  by  advising  projects  how  to  prepare  for 
OT&E.  The  Director  General  Naval  Engineering  Requirements  (DGNER)  is 
currently  responsible  for  the  design,  development,  acceptance  into  service  and 
through  life  support  of  ships.  This  engineering  policy  directorate  appears  to  be  well 
suited  to  the  DT&E  policy  and  review  function. 
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4.  OT&E  Tester  &  Evaluator 

Ideally,  the  OT&E  Tester  &  Evaluator  should  be  independent  and  focus 
exclusively  on  OT&E.  He  should  also  have  operational  credibility  and  influence  in 
operational  matters  relating  to  OT&E.  The  current  OT&E  Tester  and  Evaluator, 
Commander,  Test  and  Evaluation  (CTE),  within  the  Maritime  Command  is 
independent  from  developing  and  production  agencies,  but  is  responsible  to  the  end 
user  of  the  system.  This  arrangement  has  advantages  to  CTE  of  access  to 
knowledgeable  and  experienced  fleet  staff  warfare  officers  and  fleet  planners  for 
assistance  in  trials  planning  and  assessment.  However,  there  are  possible 
disadvantages  to  this  lack  of  independence  from  the  end  user.  The  Maritime 
Commander  may  have  aims  and  objectives  which  could  conflict  with  the  total 
impartiality  of  OT&E  reporting.  As  the  Maritime  Commander  establishes 
scheduling  priorities  for  his  resources,  priority  given  to  OT&E  is  subject  to  the 
politics  of  requirements  and  funding.  OT&E  may  take  a  low  priority  behind 
operations  and  training.  The  Maritime  Ccmimander  may  also  not  wish  to  “stress”  the 
system  during  OT&E,  in  case  he  perceives  that  operational  deficiencies  so 
highlighted  may  be  seen  as  having  some  reflection  on  his  own  organisation. 

CTE  is  currently  multifunctional,  conducting  DT&E  and  PAT&E  in 
addition  to  OT&E.  This  results  in  a  lack  of  differentiation  within  CTE’s  organisation 
between  DT&E/PAT&E  and  OT&E.  Also  being  placed  under  the  inservice 
engineering  area  of  the  Maritime  Command,  he  lacks  visibility  and  influence  within 
the  warfare  community  and  hence  credibility  in  operational  matters  critical  to 
OT&E.  CTE’s  OT&E  responsibilities  are  overshadowed  by  his  DT&E/PAT&E 
duties. 

To  achieve  the  advantages  of  DT&E  and  OT&E  separation,  a  revised 
organisation  is  proposed,  similar  to  that  outlined  by  the  former  RANTAU  in 
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Reference  63.  All  CTE’s  current  DT&E/PAT&E  responsibilities,  and  those  already 
the  responsibility  of  CSO(E),  would  remain  with  an  in-service  trials  team  under 
CSO(E).  This  trials  team,  headed  by  an  engineering  branch  officer,  would  be  formed 
from  the  current  CTE  technical  personnel,  and  could  also  conduct  certifications. 
Light  Off  Examinations,  and  the  technical  aspects  of  Ship  Qualification  Trials.  The 
team  would  provide  technical  assistance  and  advice  to  a  separate  OT&E  trials  team. 

CTE’s  OT&E  duties  would  be  removed  and  managed  by  a  separate  group. 
Commander,  OT&E  (COT&E),  working  under  an  operational  authority.  There  are 
two  such  authorities  within  the  Maritime  Command;  the  Chief  Staff  Officer 
(Operations)  (CSO(O))  who  is  responsible  for  plans  and  operations,  and 
Commodore  Flotillas  (COMFLOT)  who  is  responsible  for  Fleet  readiness.  As 
OT&E  is  the  assessment  of  operational  effectiveness  and  suitability,  COT&E  should 
be  responsible  to  COMFLOT.  Headed  by  a  Warfare  branch  officer,  COT&E  would 
manage  the  post  delivery  trials  period  for  all  ships  from  the  MHQ  perspective.  He 
would  still  manage  the  SQT,  which,  although  not  true  OT&E,  still  has  some 
operational  input  which  could  not  be  met  by  the  in-service  trials  team  under 
CSO(E).  He  would  be  the  OT&E  authority  with  responsibilities  to  manage  all  RAN 
OT&E  including  OT-3,which  is  now  only  conducted  on  first  of  class. 

5.  OT&E  Organisation  Summary 

This  proposed  reorganisation  achieves  the  aim  of  DT&E/OT&E  separation 
with  some  concessions  necessary  due  to  manpower  constraints.  Also,  by  including 
OT&E  under  COMFLOT  and  managed  by  a  Warfare  Officer,  OT&E  could  achieve 
the  visibility,  credibility  and  influence  necessary  within  the  RAN. 
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D.  RAN  OT&E  POLICY  IMPLEMENTATION 

1.  OT&E  Policy  Documentation 

RAN  T&E  documentation  needs  to  be  clear,  concise  and  comprehensive  so 
that  is  serves  as  a  valid  reference  and  guide  for  all  personnel  involved  with  T&E.  To 
correct  the  current  inconsistent  and  out  of  date  documentation,  a  major  rewrite, 
rather  than  just  updating  is  required.  This  rewrite  of  T&E  documentation  could  be 
the  vehicle  by  which  the  T&E  system  within  the  RAN  is  overhauled.  Before  the 
documentation  is  revised,  T&E  definitions  and  terminology  need  to  agreed  within 
the  RAN  and  DoD.  Then  the  OT&E  and  DT&E  organisations  within  the  RAN  need 
to  be  resolved,  and  the  appropriate  policies  decided.  This  could  be  achieved  by  the 
development  of  a  revised  RAN  T&E  DI(N),  followed  by  ABR  1921which  would  be 
firmly  based  on  the  T&E  DI(N). 

2.  Independence  of  OT&E  Agency 

Given  that  the  proposed  COT&E  is  organisationally  independent  from  the 
developing  and  production  agencies,  his  procedural  independence  should  also  be 
achieved  to  ensure  impartiality  and  validity  d*  testing.  When  OT&E  is  conducted,  it 
should  not  be  influenced  by  the  Project  or  DAA.  No  Project  or  DAA  personnel 
should  be  present  during  testing.  A  final  report  only  should  be  issued  by  the 
Maritime  Commander,  with  no.  progress  or  interim  report  (unless  the  testing  is 
unusually  lengthy),  to  allow  full  analysis  and  evaluation  of  the  test  results.  This 
applies  to  OT&E  conduct  and  reporting  only.  As  the  SQT  period  is  classified  as 
PAT&E  it  should  continue  to  be  conducted  with  the  cooperation  and  assistance  of 
Project  authorities.  Similarly,  to  ensure  test  validity  and  impartiality,  contractors 
should  not  be  present  during  OT&E. 
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3.  T&E  Planning 

The  RAN  has  embraced  the  U.S.  TEMP  concept  as  the  single  executive 
document  for  the  management  of  T&E  for  major  acquisitions.  Although  required 
eariy  in  a  project,  recent  TEMPs  have  not  been  raised  until  after  project  funding  has 
been  approved.  TEMPS  need  to  be  approved  earlier  to  enable  inclusion  of  post 
delivery  PAT&E/DT&E  and  OT-3  in  the  Project  Budget,  and  OT-4  in  MHQ  budget. 
The  authority  for  DCNS  to  release  the  TEMP  was  based  on  his  role  as  Project 
Sponsor.  With  HQADF  now  assuming  this  role,  the  DCNS  function  in  TEMP 
development  is  unclear.  The  TEMP  review  process  should  be  reviewed  as  part  of  the 
proposed  OT&E  system. 

A  T&E  Planning  Group  (TEPG)  is  currently  formed  prior  to  TEMP 
development  under  the  leadership  of  the  T&E  manager  for  the  project  to  analyse 
T&E  requirements  and  estimate  resources  required.  The  TEPG  is  similar  to  the  USN 
TECG  with  one  significant  difference.  The  USN  TECG  is  led  by  the  Director,  Test, 
Evaluation  and  Technology  Requirements  (N09I)  under  the  CNO,  whereas  the 
RAN  TEPG  is  led  by  each  individual  Project  Director  through  this  T&E  manager. 
Despite  their  best  intentions.  Project  Directors  are  essentially  driven  by  cost  and 
schedule  considerations,  not  the  overall  T&E  adequacy  of  their  project.  To  be  truly 
objective,  each  TEPG  should  be  chaired  by  the  authority  responsible  for  ensuring 
the  adequacy  of  the  RAN’s  overall  T&E  program.  The  proposed  RAN  OT&E  policy 
maker  and  overseer,  DGNPW’s  organisation,  may  best  be  suited  to  the  chairmanship 
of  each  TEPG. 

4.  Combined  DT&E  and  OT&E 

With  the  recognition  of  the  differences  between  DT&E/PAT&E  and  OT&E 
the  RAN  can  now  appreciate  the  advantages  and  difficulties  of  combined  DT&E/ 
PAT&E  and  OT&E.  By  classifying  SQT  as  PAT&E,  and  conducting  a  separate  OT- 
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3  period  for  first  of  class,  the  conduct  of  combined  and/or  concurrent  DT&E/PAT«feE 
and  OT&E  can  be  addressed  on  the  requirements  of  each  individual  trial. 

5.  Foreign  Weapons  Evaluation 

Despite  the  most  recent  Australian  Government  “White  Paper”  on  Defence 
Policy  [Ref.  6:p.  70]  requiring  the  need  to  be  able  to  determine  the  performance  in 
the  Australian  environment  of  equipment  of  both  overseas  and  local  origin,  no  DoD 
or  RAN  T&E  policy  specifically  addresses  the  evaluation  of  foreign  systems.  The 
proposed  revision  of  the  OT&E  system,  and  consequent  DT&E  revision,  would 
result  in  a  more  credible  and  useful  organisation.  Policies  could  then  be  developed 
regarding  OT&E  and  DT&E  in  the  evaluation  of  foreign  weapons  systems.  The 
current  process  of  evaluations  of  foreign  weapons  by  adhoc  Project  Teams  could  be 
modified  to  include  greater  formal  involvement  by  T&E  authorities. 

6.  Land  Based  Test  Sites 

LBTS  can  be  used  to  highlight  problems  of  operational  effectiveness  and 
suitability  early  in  system  development.  Although  LBTS  are  part  of  both  recent 
major  RAN  combatant  {xojects,  the  condiKt  of  OT&E  in  these  sites  is  planned  to  be 
limited  to  a  non-interference  basis  with  system  development  and  integration.  OT&E 
needs  to  be  conducted  during  a  dedicated  period  in  the  LBTS  following  successful 
DT&E,  to  gain  an  estimate  of  the  potential  operational  effectiveness  and  suitability 
of  the  system.  Agsun,  the  need  for  early  planning  to  include  OT&E  in  schedules  and 
funding  is  highlighted. 

7.  Whole  Ship  Testing 

The  current  OT&E  to  support  an  Acceptance  into  Naval  Service  (AINS) 
decision  (OT-3)  is  little  more  than  an  extended  SQT  period  where  DT&E/R\T&E  is 
conducted  with  some  operational  assessment.  It  is  not  considered  to  be  adequate  to 
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assess  the  first  of  a  new  ship  class  against  the  user  requirement.  To  assess  the 
operational  effectiveness  and  suitability  of  a  ship  against  the  user  requirement  with 
the  degree  of  confidence  required  for  the  Acceptance  Board  to  support  an  AINS 
decision,  a  dedicated,  free  play,  scenario  based,  OT-3  period  is  required.  The 
proposed  Delivery  to  AINS  trials  schedule,  which  is  based  on  that  currently 
proposed  for  HMAS  COLLINS  is  shown  in  Figure  27. 


Figure  27:  Proposed  Delivery  to  AINS  Trials  Schedule 

OT-B  should  be  conducted  after  the  ship  has  completed  the  SQT  and 
Operational  Readiness  Evaluation  to  ensure  that  the  testing  is  conducted  on  a 
worked  up  and  materially  proven  ship.  To  be  effective  the  testing  should  be 
conducted  of  the  overall  ship  as  a  complete  weapons  system.  The  recent  USN 
innovation  of  ‘‘whole  ship  testing”  provides  a  good  example  of  the  testing  proposed. 
Whole  ship  testing  includes  engineering  and  technical  problems  combined  with 
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combat  system  activity.  It  emphasises  operational  realism  by  testing  the  ship  as 
though  in  war,  and  is  designed  to  provide  a  realistic  threat  in  an  operational 
environment,  and  an  element  of  surprise  to  provide  realistic  tests  to  support 
completion  of  the  OT&E  objectives  and  resolutions  of  the  COIs. 

The  OT-4  period  leading  to  Operational  Acceptance  (OA)  proposed  in  this 
OT&E  model  is  also  based  on  the  schedule  currently  proposed  by  CTE  for  HMAS 
COLLINS.  An  example  of  an  AINS  to  OA  trials  schedule  is  shown  in  Figure  28. 


Figure  28:  Proposed  AINS  to  OA  Trials  Schedule 


It  includes  a  long  term  RM&A  and  ILS  evaluation,  further  testing  to 
achieve  statistically  significant  results  in  critical  areas  and  the  assessment  of  the 
operational  effectiveness  and  operational  suitability  of  the  ship  in  a  major  exercise 
and  on  deployment. 


185 


E.  RAN  SHIP  ACCEPTANCE 

The  Acceptance  Board,  with  its  broader  experience  and  knowledge  than  Trials 
Authority,  is  generally  agreed  within  the  RAN  as  being  successful  as  a  board  of 
review  of  the  planning  and  results  of  T«feE  for  individual  projects,  and 
recommending  PA  and  AINS.  As  an  ad-hoc  organisation  consisting  of  various 
functional  specialists  brought  together  part  time,  however,  the  AB  suffers  from  a 
number  of  problems. 

The  AB  requires  guidance  as  to  its  role  and  functions,  T&E  philosophy  and  the 
procedures  of  the  acceptance  process.  Although  the  AB  attempts  to  obtain  sufficient 
guidance  from  the  T&E  documentation,  it  has  found  this  documentation  to  be 
unclear  and  conflicting.  To  enable  AB  consistency  and  development,  the  Board 
needs  an  administrator,  or  permanent  VP.  RANTAU  was  the  administrator  and 
provided  guidance  to  the  Board,  with  CTE  now  assuming  this  role.  However,  the 
AB  should  be  independent  from  the  T&E  authority  as  it  assesses  the  trials  planned 
and  conducted  by  that  authority. 

With  only  DT&E/PAT&E  and  the  minimal  operational  aspects  of  the  previous 
OT-3  testing  conducted  by  the  RAN,  the  AB  have  tended  to  focus  on  materiel 
deficiencies.  With  a  more  comprehensive  OT-3  period  proposed  to  be  conducted, 
the  AB  members  could  more  effectively  use  their  knowledge  and  experience  in 
assessing  the  results  of  OT&E  against  the  user  requirement  rather  than  on 
workmanship  and  materiel  deficiencies. 

The  AB  should  be  convened  for  the  first  of  class  only,  providing  the  specialist 
expertise  to  interpret  requirements  and  determine  the  level  of  acceptability  required 
for  AINS.  The  AB  consists  of  Naval  Officers  drawn  part-time  from  their  normal 
jobs,  and  so  the  cost  appears  to  be  minimal.  The  AB,  however,  has  a  high 
opportunity  cost,  which  is  the  value  of  the  next  best  alternative  on  which  the 
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members  could  be  working  instead  of  serving  on  the  Board.  Once  the  first  of  class 
is  accepted  the  Board’s  task  is  complete.  The  interpretations  and  levels  of 
acceptability  established  by  the  AB  then  guide  the  trials  authority  in  recommending 
acceptance  of  each  of  the  remaining  ships. 

F.  RAN  OT&E  PROCEDURES 

1.  Assessment  of  Operational  Effectiveness 

With  the  proposed  inclusion  of  a  dedicated  OT-3  period,  the  OT&E 
authority  will  need  to  develop  tests  for  a  free  play,  scenario  based  environment 
specifically  designed  to  assess  the  COIs  against  the  User  Requirements.  This  could 
be  achieved  along  USN  lines  where  each  COI  and  its  related  operational 
effectiveness  objective  is  examined  and  the  information  needed  to  be  known  to 
enable  each  objective  to  be  assessed  is  determined.  The  test  design  would  have  to 
take  into  consideration  data  collection  methods  and  range  considerations.  The 
analysis  of  these  tests  may  be  different  from  the  weapons  analysis  currently 
performed.  Current  weapons  analysis  focuses  on  individual  weapon  system 
performance,  usually  from  “fire  push”  to  “weapon  destruct”,  whereas  whole  ship 
testing  in  a  free  play  environment  would  require  a  broader  analysis.  An  increased  of 
use  modeling  &  simulation  may  also  be  required  to  assess  some  systems  in  a  cost 
effective  manner.  It  is  beyond  the  scope  of  this  thesis  to  investigate  these 
requirements,  however,  close  liaison  between  the  OT&E  authority  and  the  RAN’s 
current  analysis  groups  will  be  required  to  achieve  valid  testing  and  subsequent 
analysis. 

2.  Assessment  of  Operational  Suitability 

The  current  operational  suitability  areas  now  assessed  are  considered  valid. 
These  are  listed  in  Table  23. 


187 


TABLE  23:SUITABILITY  ISSUES  FOR  ASSESSMENT 
Suitability  Issues 

Availability 

Reliability 

Maintainability 

Safety 

Human  Factors 
Interoperability 
Compatibility 

Integrated  Logistic  Support  including: 

Maintenance  Planning 

Supply  Support 

Support  and  Test  Equipment 

Technical  E>ata  and  Documentation 

Manpower  and  Personnel 

Training  and  Training  Devices 

Facilities 

Packaging,  Handling,  Storage  and  Transport 
Computer  Support 
Configuration  Control 


The  assessment  of  operational  suitability,  especially  for  the  RAN’s  current 
combatant  projects,  should  focus  on  interoperability  and  compatibility  as  these 
present  the  highest  risks  with  the  integration  of  proven  equipment  in  a  unique 
combination  in  a  ship  platform. 

The  assessment  of  ILS  should  continue  to  be  conducted  in  two  phases: 

•  ILS  Structural  Review.  Conducted  prior  to  operational  testing,  this  review 
assesses  if  the  ILS  intended  to  provided  will  support  the  ILS  policy  of  the 
Project 

•  ILS  Operational  Review.  During  the  operational  testing  period,  the  ship's 
ILS  related  documentation  and  records  are  reviewed,  maintainability 
demonstrations  are  performed  and  a  supportability  assessment  is  conducted. 
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RM&A  data  is  collected  and  analysed  during  OT-3  to  give  an  indication  of 
performance  prior  to  AINS,  and  then  continued  for  12  months  during  the  OT-4 
phase.  The  data  should  continue  to  be  collected  and  analysed  on  an  “as  required”, 
vice  continuous,  basis  by  the  Trials  Authority  until  the  RAN  develops  its  routine  in- 
service  RM&A  data  collection  and  analysis  system.  The  analysis  of  RM&A  data  is 
a  specialist  task,  currently  outside  the  experience  of  most  uniformed  personnel.  To 
achieve  significant  and  valid  results,  an  RM&A  analysis  capability  needs  to  be 
established,  possibly  within  the  Maritime  Command’s  weapons  assessing  area. 

3.  OT&E  Personnel,  Training  and  Career  Management 

An  effective  OT&E  system  is  dependent  on  knowledgable  and  experienced 
personnel.  OT&E  is  a  specialised  discipline,  with  its  own  philosophy  and 
methodology,  however,  very  few  personnel  have  had  any  T&E  experience 
knowledge  prior  to  joining  the  OT&E  authority.  This  makes  training  important,  but 
the  RAN  has  no  training  courses  on  OT&E,  or  on  T&E  in  general.  A  number  of 
officers  have  completed  the  USN  Operational  Test  Directors  Course  in  recent  years 
which  has  proved  beneficial  to  the  knowledge  and  understanding  of  OT&E  within 
CTE.  This  course  should  be  a  pre-requisite  for  key  billets  within  the  OT&E  authority 
until  the  RAN  develops  enough  knowledge  and  expertise  to  develop  its  own  course. 
Although  the  USN  OTD  course  is  only  of  a  weeks’  duration,  it  is  understood  that  the 
USN  is  investigating  a  more  comprehensive  course.  Other  authorities  also  require 
training  in  OT&E.  The  OT&E  authority  needs  to  develop  acquaint  courses  for  other 
personnel  with  a  need  to  understand  OT&E  e.g..  Project  T&E  managers  and 
capability  requirements  developers.  Also  to  increase  the  awareness  of  OT&E  in  the 
RAN,  the  OT&E  authority  needs  to  deliver  presentations  regularly  to  Project 
Directors  Courses,  the  ILS  Management  and  Acquisition  Course  (ILSMAC), 
Defence  management  courses,  and  other  appropriate  acquisition  courses. 
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Although  some  personnel  may  develop  into  OT&E  specialists,  no  career 
path  in  OT&E  is  provided  in  the  RAN.  The  warfare  officer  functional  heads  within 
the  OT&E  authority  cannot  currently  aspire  to  command  of  the  authority  as  that 
position  is  currently  a  Weapons  Electrical  Engineer’s  billet.  The  proposed  change  of 
the  head  of  the  OT&E  authority  to  a  Warfare  Officer  as  COT&E  should  provide  a 
career  path  within  the  authority.  As  the  credibility  of  OT&E  increases,  the 
knowledge  and  experience  gained  by  warfare  officers  in  this  area  will  be  seen  to  be 
of  more  benefit  in  future  warfare  related  positions,  thus  enabling  a  longer  term  career 
path.  The  proposed  employment  of  COT&E  under  COMFLOT  should  also  be  more 
career  enhancing  for  operations  personnel  than  the  current  arrangement  under 
CSO(E). 

The  professionalism  and  knowledge  of  T&E  personnel  could  be  enhanced 
by  membership  in  a  related  professional  organisation.  The  International  Test  and 
Evaluation  Association  (ITEA)  is  one  such  professional  organisation. 
Headquartered  in  the  U.S.  with  a  chapter  in  Australia,  the  association  encourages  the 
development  and  exchange  of  technical  information  in  the  field  of  test  and 
evaluation.  The  corporate  membership  of  a  suitable  professional  organisation  may 
be  worthwhile  to  the  RAN. 

G.  OT&E  MODEL  SUMMARY 

The  basis  for  this  OT&E  model  are  clear  definitions  of  the  types  and  phases  of 
T&E.  The  current  definitions  are  amended  and  phases  of  T&E  are  refined  to  better 
suit  the  Australian  Defence  environment.  These  refined  T&E  phases  are 
incorporated  into  an  amended  TSTP,  termed  collectively  as  “Integrated  Tests  and 
Trials”,  which  recognises  the  delineation  between  DT&E  and  OT&E.  No  change  is 
suggested  to  the  current  relationship  between  OT&E  phases  and  acquisition 
milestones. 
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The  current  OT&E  organisation  structure  is  modified  to  enable  effective  OT&E 
to  support  the  acquisition  process.  It  provides  for  authorities  responsible  for  the 
management,  policy  making  and  oversight,  and  specification  of  OT&E.  This 
proposed  reorganisation  closely  achieves  the  aim  of  DT&E/OT&E  separation. 

The  model  proposes  the  inclusion  of  a  dedicated  OT-3  period  in  a  free  play, 
scenario  based  environment.  The  assessment  of  operational  effectiveness  is 
specifically  designed  to  assess  the  COIs  against  the  User  Requirements  and  will 
require  a  different  approach  to  analysis.  The  assessment  of  operational  suitability 
focuses  on  interoperability  and  compatibility,  and  the  assessment  of  ILS  is  achieved 
using  current  methods,  but  with  improved  RM&A  analysis.  This  enhanced  OT&E 
and  minor  modifications  to  the  AB  should  provide  CNS  with  a  more  comprehensive 
and  thorough  recommendation  for  AINS. 

To  be  effective,  OT&E  ideally  requires  current,  experienced,  knowledgable 
personnel  with  high  professional  credibility  within  their  field  of  expertise.  To 
achieve  this,  the  model  includes  adequate  training,  good  guidance  documentation, 
and  a  career  path  where  OT&E  is  seen  as  career  progression  by  capable  personnel. 

This  OT&E  model  for  the  RAN  uses  principles  derived  from  the  USN  system. 
By  taking  consideration  of  resource  limitations,  characteristics  and  culture  of  the 
RAN,  it  establishes  an  effective  OT&E  organisation  which  is  important  to  the 
testing,  evaluation  and  subsequent  acceptance  decision  of  the  new  combatants. 
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VII.  CONCLUSIONS 


A.  GENERAL  CONCLUSION 

The  USN  OT&E  system  has,  by  necessity,  developed  into  a  well  organised, 
well  documented  and  effective,  if  complex,  system.  The  RAN  system,  on  the  other 
hand,  suffers  from  conflicting  and  confusing  T&E  policy  documentation,  a  weak 
OT&E  organisation  structure  and  a  general  lack  of  OT&E  knowledge  and 
appreciation.  The  determination  of  OT&E  required  has  traditionally  not  been 
achieved  early  enough  in  the  life  of  a  project,  and  so  project  funding  has  not  included 
the  provision  for  OT&E.  The  RAN  OT&E  system,  although  originally  based  on 
USN  OT&E  philosophy,  now  differs  in  the  importance,  interpretation,  application 
and  focus  of  OT&E.  As  a  result  of  organisation  changes,  it  now  lacks  the  ability  to 
effectively  manage  overall  T&E,  let  alone  OT&E,  at  a  time  when  the  operational 
risk  of  ship  projects  is  increasing.  To  achieve  efficient  and  effective  trials  and 
acceptance  of  the  new  higher  risk  warships  currently  under  construction,  the  RAN 
OT&E  system  needs  to  be  revised. 

R  SPECIFIC  CONCLUSIONS 

1.  OT&E  in  the  Acquisition  Process 

Australia’s  Defence  strategy  has  progressed  from  a  position  of  dependence 
on  allies,  to  a  positive  acceptance  of  both  self-reliance  and  regional  infl  uence.  To 
achieve  its  Defence  goals,  however,  Australia  devotes  much  less  of  its  resources  and 
[daces  less  priority  on  R&D  than  does  the  U.S.  The  need  for  local  T&E  is  recognised 
by  the  Australian  Government  and  DoD,  for  both  indigenous  and  overseas  systems, 
however,  the  importance  (rf  OT&E  in  particular,  is  not  specifically  addressed  in  any 
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Australian  T&E  policy  document.  This  lack  of  recognition  of  the  importance  of 
OT&E  is  perhaps  due  to  the  RAN  procuring  low  risk,  complete  ships  in  the  past 
(e.g.,  FFG-7  class)  with  the  wealth  of  USN  DT&E  and  OT&E  behind  it. 

U.S.  OT&E  is  viewed  as  being  more  important  as  a  basis  for  a  decision  to 
proceed  beyond  Low  Rate  Initial  Production,  than  on  the  introduction  into  service 
of  the  final  production  item.  This  is  because  the  “big  bucks”  of  most  Defence 
acquisitions  are  spent  during  the  production  phase.  OT&E,  however,  has  started  to 
play  an  important  role  also  in  the  assessment  of  first  of  class  systems  for  introduction 
into  the  USN  Fleet. 

2.  OT&E  Organisation 

The  RAN  and  Australian  DoD  have  reorganised  over  the  past  five  years  to 
create  a  more  effective  and  efficient  user  requirements  organisation,  and  to  better 
accord  with  Program  Management  and  Budgeting  principles.  However,  as  a  result, 
the  RAN  has  no  T&E,  or  OT&E  policy  maker  and  overseer,  no  OT&E  coordinator 
within  Navy  Office,  and  the  RAN’s  OT&E  authority  is  buried  within  the  in-service 
engineering  management  area  of  the  Maritime  Command. 

3.  OT&E  Policy 

Where  USN  OT&E  policy  has  its  basis  in  the  statutes  approved  by 
Congress,  the  RAN  has  no  such  formal  basis  and  its  OT&E  policy  is  embedded  in 
the  more  general  T&E  policy.  The  RAN  has  adapted  the  more  detailed  USN  T&E 
definitions  to  suit  its  own  requirements.  During  this  process,  however,  the  RAN  lost 
the  distinction  between  the  types  of  T&E,  and  often  confuses  OT&E  with  PAT&E/ 
DT&E.  Overall,  RAN  T&E  documentation  is  not  comprehensive,  inconsistent  and 
fails  to  provide  adequate  guidance. 

The  Commander,  Test  and  Evaluation  (CTE),  on  the  staff  of  the  Maritime 
Commander,  is  independent  from  the  development  and  production  agencies,  but  not 
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from  the  end  user,  as  in  the  USN.  The  implications  of  this  lack  of  independence  is 
that  the  end  user  may  have  his  own  aims  and  objectives  which  conflict  with  the  total 
impartiality  of  OT&E  conduct  and  reporting. 

E>espite  the  most  recent  Australian  Government  “White  Paper”  on  Defence 
Policy  requiring  the  need  to  be  able  to  determine  the  performance  in  the  Australian 
environment  of  equipment  of  both  overseas  and  local  origin,  no  DoD  or  RAN  T&E 
policy  specifically  addresses  the  evaluation  of  foreign  systems. 

Although  used  primarily  for  the  purpose  of  conducting  DT&E,  the  USN  has 
demonstrated  that  early  OT&E  on  Land  Based  Test  Sites  (LETS)  can  give  an 
estimation  of  potential  operational  effectiveness  and  suitability,  and  hence  identify 
potential  operational  problems  early  and  minimise  operational  risk.  Although  LBTS 
are  being  developed  for  both  the  ANZAC  frigate  and  Collins  submarine  projects,  the 
use  of  these  LBTS  for  early  OT-3  was  not  planned  in  the  original  project  schedules, 
and  so  is  now  subject  to  them  being  used  on  a  non-interfering  basis  to  the  contractor. 

The  RAN  does  not  clearly  differentiate  between  the  types  of  T&E,  which 
leads  to  DT&E  and  OT&E  often  being  conducted  over  the  same  period,  by  the  same 
test  team.  Although  the  RAN  recognises,  by  definition,  the  difference  in  objectives 
and  methodology  between  the  types  of  T&E,  no  limitations  or  guidance  as  to  the 
possible  hazards  of  this  combined  testing  approach  are  addressed. 

4.  Ship  Acceptance 

As  an  ad-hoc  organisation  consisting  of  various  functional  specialists 
brought  together  part  time,  the  Acceptance  Board  (AB)  suffers  from  a  number  of 
problems.  It  requires  guidance  as  to  its  role  and  functions,  T&E  philosophy  and  the 
procedures  of  the  acceptance  process.  Although  the  AB  attempts  to  obtain  sufficient 
guidance  from  the  T&E  documentation,  it  has  found  this  documentation  to  be 
unclear  and  conflicting.  With  only  DT&E/PAT&E  and  the  minimal  operational 
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aspects  of  the  OT-3  testing  conducted  by  the  RAN,  the  AB  has  tended  to  focus  on 
materiel  deficiencies,  rather  than  the  performance  of  the  first  of  class  against  the  user 
requirement. 

5.  OT&E  Procedures 

The  RAN  recognises  that  the  user  requirement  is  the  bench  mark  for 
determining  the  degree  to  which  a  system  is  effective,  and  also  that  operational 
effectiveness  is  best  assessed  by  a  performance  demonstration  by  normal  operating 
personnel  in  the  normal  or  given  environment.  Assessing  operational  effectiveness 
has  involved  analyzing  each  Critical  Operational  Issue  and  then  employing  modified 
Ship  Qualification  Trial  or  Fleet  Exercise  Program  techniques  to  evaluate  them.  The 
RAN  has  very  little  guidance  on  the  assessment  of  operational  effectiveness. 

The  principles  of  operational  suitability  assessment  are  common  between 
the  USN  and  RAN.  Although  assessments  by  the  RAN  have  become  more 
subjective  over  recent  years,  the  assessment  of  RM&A  is  still  in  its  infancy.  The 
RAN  requires  experienced  personnel  to  design  suitability  tests  and  analyse  results. 

The  authority  for  the  Deputy  Chief  of  Naval  Staff  (DCNS)  to  release  the 
Test  and  Evaluation  Master  Plan  (TEMP)  was  based  on  his  role  as  Project  Sponsor. 
Now  with  HQADF  assuming  this  role,  the  DCNS  function  in  TEMP  development  is 
unclear.  The  TEMP  approval  process  requires  review. 

The  RAN  Test  and  Evaluation  Planning  Group  (TEPG)  for  each  major 
project  is  chaired  by  each  individual  Project  Director  through  his  T&E  manager. 
Despite  their  best  intentions.  Project  Directors  are  essentially  driven  by  cost  and 
schedule  considerations,  not  the  overall  T&E  adequacy  of  their  project.  To  be  truly 
objective,  each  TEPG  should  be  chaired  by  the  authority  responsible  for  ensuring 
the  adequacy  of  the  RAN’s  overall  T&E  program. 


The  eariy  dissemination  of  OT&E  results,  before  a  full  analysis  is  complete 
and  the  implications  assessed,  can  lead  to  other  authorities  talcing  hasty  action  on 
incomplete  information,  and  can  prejudice  the  OT&E  authority’s  final  conclusions 
and  recommendations.  OT&E  results  should  indicate  system  deficiencies  against 
user  requirements  rather  than  equipment  defects,  so  the  results  need  to  be  fully 
analysed  and  the  implications  assessed  before  being  reported  to  a  wider  audience. 

Within  the  RAN,  the  current  OT&E  to  support  an  AINS  decision  (OT-3)  is 
little  more  than  an  extended  SQT  period  where  DT&E/PAT&E  is  conducted  with 
some  operational  assessment.  It  is  not  considered  to  be  adequate  to  assess  the  first 
of  a  new  ship  class  against  the  user  requirement. 

To  be  effective,  OT&E  requires  experienced  and  knowledgable  personnel 
with  high  professional  credibility  within  their  field  of  expertise.  To  achieve  this 
requires  selection  of  suitable  personnel,  adequate  training  and  good  guidance 
documentation,  and  preferably  a  career  path  where  serving  in  the  OT&E  Authority 
is  seen  as  career  progression  by  such  capable  personnel. 
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VIII.  RECOMMENDATIONS 


The  OT&E  model  recommended  for  the  RAN  uses  principles  derived  from  the 
USN  system.  By  taking  consideration  of  resource  limitations,  and  the  characteristics 
and  culture  of  the  RAN,  it  establishes  an  effective  OT&E  organisation  which  is 
important  to  the  testing,  evaluation  and  subsequent  acceptance  decision  of  the  new 
combatants. 

A.  GENERAL  RECOMMENDATIONS 

It  is  recommended  that  the  proposed  OT&E  model  be  reviewed,  evaluated  and 
implemented  in  the  RAN  to  achieve  an  effective  OT&E  system  to  support  the 
introduction  of  operationally  effective  and  operationally  suitable  new  combatants 
into  the  RAN. 

R  SPECinC  RECOMMENDATIONS 

1.  OT&E  Deflnitions 

It  is  recommended  that: 

a.  the  current  definitions  and  phases  of  ship  T&E  be  amended  to  better  suit 
the  Australian  Defence  environment. 

b.  the  current  TSTP  be  revised  to  show  DT&E  and  OT&E  events  as 
separate  categories  of  testing,  and  these  trials  be  termed  collectively  as 
Integrated  Tests  and  T rials  (ITT). 
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2.  OT&E  Documentation 

It  is  recommended  that: 

a.  the  RAN  T&E  DI(N)  and  ABR  1921  be  revised  to  achieve 
comprehensive  and  consistent  T&E  policy  guidance. 

3.  OT&E  in  the  Acquisition  Process 

It  is  recommended  that: 

a.  OT&E  be  conducted,  where  possible,  prior  to  the  production  or 
purchase  decision  in  the  acquisition  process  to  minimise  risk. 

b.  OT-3  be  separated  from  the  DT&E  /PAT&E  post  delivery  trials. 

c.  OT-3  be  conducted  as  a  dedicated,  free  play,  scenario  based  period  after 
the  completion  of  SQT  and  ORE  to  ensure  testing  is  conducted  on  a 
worked  up  and  materially  proven  ship. 

d.  OT-3  of  the  overall  ship  be  conducted  as  a  complete  weapons  system. 

4  OT&E  Organisation 

It  is  recommended  that: 

a.  the  appointment  of  a  DoD  OT&E  policy  maker  and  overseer  within 
HQADF  be  investigated. 

b.  DGNPW  be  appointed  as  the  OT&E  policy  maker  and  overseer  within 
the  RAN. 

c.  a  close  working  relationship  on  T&E  matters  be  required  between 
DGFEHSea),  DGNPW  and  the  OT&E  authority. 

d.  DGNER  be  appointed  as  the  Materiel  Division  T&E  agencv  to  assist 
Project  Directors  in  complying  with  policies,  incorporate  lessons 
learned  from  previous  projects,  review  the  T&E  aspects  of  projects  at 
major  milestones,  and  advise  projects  how  to  prepare  for  OT&E. 
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e.  the  current  GTE  organisation  be  disestablished. 

f.  an  “In-service  Trials”  team  be  established  under  CSO(E)  to  conduct 
CTE’s  current  DT&E/PAT&E  responsibilities,  and  those  already  the 
responsibility  of  CSCXE)- 

g.  Commander,  OT&E  (COT&E)  be  established  as  the  RAN’s  OT&E 
authority  under  COMFLOT,  to  manage  OT&E  and  the  post  delivery 
trials  period  for  all  ships  from  the  MHQ  perspective. 

5.  OT&E  Policy 

It  is  recommended  that: 

a.  policy  be  developed  to  enable  the  performance  in  the  Australian 
environment  of  foreign  equipment  to  be  evaluated  efficiently, 
effectively  and  consistently. 

b.  the  TEMP  approval  process  be  reviewed  to  ensure  appropriate 
authorities  are  consulted  and  approve  the  TEMP. 

c.  the  TEPG  be  chaired  by  DGNPW  as  the  authority  appointed  to  ensure 
the  adequacy  of  the  RAN’s  overall  T&E  program. 

d.  OT&E  not  be  influenced  by  the  Project  or  DAA,  and  that  Project  or 
DAA  personnel  not  be  present  during  testing. 

e.  contractors  not  be  present  during  OT&E. 

f.  TEMPs  for  non-development  projects  be  approved  before  production 
approval  to  enable  inclusion  of  post  delivery  PAT&E/DT&E  and  OT-3 
in  the  Project  Budget,  and  OT-4  in  MHQ  budget. 

g.  OT&E  be  conducted  during  a  dedicated  period  in  the  LBTS  following 
successful  DT&E,  to  gain  an  estimate  of  the  potential  operational 
effectiveness  and  suitability  of  the  system. 
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6.  Ship  Acceptance 

It  is  recommended  that; 

a.  the  Acceptance  Board  remain  as  a  board  of  review. 

b.  the  Acceptance  Board  be  convened  for  the  first  of  class  only. 

7.  OT&E  Procedures 

It  is  recommended  that: 

a.  The  OT&E  authority  develop  tests  for  a  free  play,  scenario  based 
environment  specifically  designed  to  assess  the  COIs  against  the  user 
requirement. 

b.  analysis  requirements  of  these  tests  be  assessed,  as  they  may  be 
different  from  the  weapons  analysis  currently  performed. 

c.  experienced  personnel  assist  in  designing  suitability  tests  and  analysing 
results. 

d  interoperability  and  compatibility  be  a  major  focus  during  the 
assessment  of  operational  suitability,  especially  for  the  RAN’s  current 
combatant  projects. 

e.  an  improved  RM&A  analysis  capability  be  established. 

f.  RM&A  data  continue  to  be  collected  and  analysed  on  an  ‘as  required’ 
basis  until  the  RAN  develops  its  routine  in-service  RM&A  data 
collection  and  analysis  system 

g.  OT&E  reports  be  made  solely  by  the  OT&E  authority. 

h.  OT&E  phase  reports  be  issued  only,  with  no  progress  or  interim  reports, 
to  allow  full  analysis  and  evaluation  of  the  test  results. 

i.  the  USN  Operational  Test  Director  type  course  be  a  prerequisite  for  key 
billets  within  the  OT&E  authority  until  the  RAN  develops  enough 
knowledge  and  expertise  to  develop  its  own  course. 
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j.  the  OT&E  authority  develop  acquaint  courses  for  other  personnel  with 
a  need  to  understand  OT&E. 

k.  the  OT&E  authority  deliver  presentations  regularly  to  Project  Directors 
Courses,  the  ILS  Management  and  Acquisition  Course  (ILSMAC), 
Defence  management  courses,  and  other  appropriate  acquisition 
courses,  so  to  increase  the  awareness  of  OT&E  in  the  RAN. 

l.  a  Warfare  Officer  be  appointed  to  head  the  OT&E  authority. 

m.  establishing  corporate  membership  of  a  suitable  professional  T&E 
organisation  be  investigated. 
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APPENDIX  A:  USN  OT&E  COURSE  SYLLABUS 


The  Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
conducts  a  four  day  Operational  Test  Directors  (OTD)  course  designed  to  acquaint 
prospective  Operational  Test  Directors  and  other  interested  personnel  with  the 
fundamentals  of  OT&E  philosophy,  terminology,  policy  and  procedures.  This 
course  covers  the  following  subject  areas: 

•  Acquisition  Program  Oven.'iew 

•  Test  and  Evaluation  Master  Plan  (TEMP) 

•  Operational  Effectiveness  Overview 

•  Operational  Suitability  Overview 

•  Logistic  Supportability 

•  Reet  Research  and  Development  T&E  Support 

•  Operationd  Security 

•  Managing  Money 

•  Weapons  Systems  Survivability 

•  Threat  Support 

•  Test  Planning 

•  Software  Intensive  Systems 

•  Modeling,  Simulation  and  Analysis 

•  Conducting  the  Tests 

•  Lessons  Learned 

•  Standards  of  Conduct 

•  Evaluation  Reports 

•  Director,  OT&E  Perspectives 

•  Chief  of  Naval  Operations  Perspectives 

•  Program  Manager  Perspectives 


•  Relationship  with  General  Accounting  Office 

•  Test  Resources 

•  Program  Formulation 
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APPENDIX  B:  ACRONYM  LIST 


ABBREVIATION  FULL  TITLE 

A 


AAW 

Antiair  Warfare 

AB 

Acceptance  Board  (RAN) 

ABR 

Australian  Book  of  Reference  (RAN) 

ACDEV-N 

Assistant  Chief  of  Naval  Staff  (Development) 

ACMAT-N 

Assistant  Chief  of  Naval  Staff  (Materiel) 

ADF 

Australian  Defence  Force  (RAN) 

AF 

Australian  Frigate  (RAN) 

AINS 

Acceptance  into  Naval  Service  (RAN) 

AMP 

Acceptance  Management  Plan  (RAN) 

ASC 

Agreed  Ship  Characteristics  (RAN) 

ASUW 

Antisurface  Warfare 

ASW 

Antisubmarine  Warfare 

C 

CAP 

Combat  Air  F^trol 

CCP 

Chief  of  Capital  Procurement  (RAN) 

CEPMAN  1 

Capital  Equipment  Procurement  Manual  (RAN) 

CIWS 

Close-in  Weapons  System 

CNO 

Chief  of  Naval  Operations  (USN) 

CNS 

Chief  of  Naval  Staff  (RAN) 

COEA 

Cost  and  Operational  Effectiveness  Analysis 
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COI 

COMFLOT 

COMNAVSEA 

COMOPTEVFOR 

CPO 

CSE 

CSCXE) 

CS0(0) 

CSSQT 

CST 

CTE 

D 

DAA 

DCNS 

DDG 

DFCP 

DGFD(SEA) 

DGNPW 

DGNW 

DI(N) 

DLSIE 

DNUR 

DNW 


Critical  Operational  Issue 
Commodore  Flotillas  (RAN) 

Commander,  Naval  Sea  Systems  Command  (USN) 
Commander,  Operational  Test  and  Evaluation 
Force  (USN) 

Capital  Procurement  Organisation  (RAN) 

Combat  System  Evaluation  (RAN) 

Chief  Staff  Officer  (Engineering)  (RAN) 

Chief  Staff  Officer  (Operations)  (RAN) 

Combat  System  Ship  Qualification  Trial  (RAN) 
Combat  System  Trainer  (RAN) 

Commander  Test  and  Evaluation  (RAN) 

Design  Approval  Authority  (RAN) 

Deputy  Chief  of  Naval  Staff  (RAN) 

Guided  Missile  Destroyer 

Defence  Force  Capability  Proposal  (RAN) 

Director  General  Force  Development  (Sea)  (RAN) 
Director  General  Naval  Policy  and  Warfare  (RAN) 
Director  General  Naval  Warfare  (RAN) 

Defence  Instruction  (Navy)  (RAN) 

Defence  logistics  Studies  Information  Exchange 
(USN) 

Director  of  Naval  User  Requirements  (RAN) 
Director  of  Naval  Warfare  (RAN) 
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DoD 

Department  of  Defence 

DOT&E 

Director  of  Operational  Test  and  Evaluation  (USN) 

DSMC 

Defence  Systems  Management  College  (USN) 

DT&E 

Development  Test  and  Evaluation 

E 

ECM 

Electronic  Countermeasures 

ESM 

Electronic  Surveillance  Measures 

EW 

Electronic  Warfare 

F 

Fcr 

Final  Contract  Trials  (USN) 

FFG 

Guided  Missile  Frigate 

FLEETEX 

Fleet  Exercise  (USN) 

FOT&E 

Follow-on  Operational  Test  and  Evaluation  (USN) 

FSD 

Full  Scale  Development  (USN) 

G 

GAO 

General  Accounting  Office  (USN) 

GDP 

Gross  Domestic  Product 

H 

HATS 

HMAS 

HQADF 

I 

ILS 

Harbour  Acceptance  Trials  (RAN) 

Her  Majesty’s  Australian  Ship 

Headquarters,  Australian  Defence  Force  (RAN) 

Integrated  Logistic  Support 

ILSP 

Integrated  Logistic  Support  Plan 

lOT&E 

Initial  Operational  Test  and  Evaluation  (USN) 
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ITP 

Integrated  Test  Package  (RAN) 

ITT 

L 

Inspections,  Tests  and  Trials  (RAN) 

LBTS 

Land  Based  Test  Site 

LRIP 

Low  Rate  Initial  Production  (USN) 

M 

MHQ 

Maritime  Headquarters  (RAN) 

MNS 

Mission  Need  Statement  (USN) 

MOE 

Measures  of  Effectiveness 

N 

NAVSEA 

Naval  Sea  Systems  Command  (USN) 

NCP 

Naval  Capability  Proposal  (RAN) 

NWAC 

Naval  Weapons  Analysis  Centre  (USN) 

O 

OA 

Operational  Acceptance  (RAN) 

OCP 

Operational  Concept  Paper  (RAN) 

OPD 

Operational  Performance  Demonstration  (RAN) 

OPEVAL 

Operational  Evaluation 

OPTEVFOR 

Operational  Test  and  Evaluation  Force  (USN) 

ORD 

Operational  Requirements  Document  (USN) 

ORE 

Operational  Readiness  Evaluation  (RAN) 

OSD 

Office  of  the  Secretary  of  Defence  (USN) 

OTA 

Operational  Test  and  Evaluation  Agency  (USN) 

OTD 

Operational  Test  Director  (USN) 

OT&E 

Operational  Test  and  Evaluation 
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PA 

Preliminary  Acceptance  (RAN) 

PAT 

Prcxluction  Acceptance  Test  (RAN) 

PAT&E 

Production  Acceptance  Test  and  Evaluation 

PD 

Project  Director  (RAN) 

PDA 

Post  Delivery  Availability 

PDT&T 

Post  Delivery  Tests  and  Trials  (RAN) 

PM 

Program  Manager  (USN) 

PMBS 

Program  Management  and  Budgeting  System 

PREINS  UR  V 

(RAN) 

President,  Board  of  Inspection  and  Survey  (USN) 

PSA 

Post  Shakedown  Availability  (USN) 

R 

R&D 

Research  and  Development 

RAN 

Royal  Australian  Navy 

RANTAU 

RAN  Trials  mid  Assessing  Unit  (RAN) 

REFTRA 

Refresher  Training  (USN) 

RM&A 

Reliability,  Maintainability  and  Availability 

ROC 

c 

Required  Operational  Characteristics  (RAN) 

o 

SATS 

Sea  Acceptance  Trials  (RAN) 

SECDEF 

Secretary  of  Defence 

SQT 

Ship  Qualification  Trial  (RAN) 

STW 

Strike  Warfare  (USN) 

SOE 

Schedule  of  Events  (USN) 

SUPSHIPS 

T 


Supervisor  of  Shipbuilding  (USN) 


T&E 

TECG 

TEMP 

TEPG 

TLR 

TSTP 

U 

USN 

uss 

w 

WSRL 


Test  and  Evaluation 

Test  and  Evaluation  Coordination  Group  (USN) 
Test  and  Evaluation  Master  Plan 
Test  and  Evaluation  Planning  Group  (RAN) 
Top  Level  Requirement  (USN) 

Total  Ship  Test  Program 

United  States  Navy 
United  States  Ship 

Weapons  Systems  Research  Laboratory  (RAN) 
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